Swin Transformer: Hierarchical ViT with Shifted Windows

TL;DR

Swin Transformer solves the fundamental scalability problem of Vision Transformers by replacing ViT's global self-attention with window-based local attention, then introducing a shifted window scheme that restores cross-window information flow. The result is a hierarchical vision backbone with linear computational complexity in image size, producing multi-scale feature maps that plug directly into existing dense prediction frameworks. It achieved state-of-the-art results on ImageNet classification (87.3% top-1), COCO object detection (58.7 box AP), and ADE20K segmentation (53.5 mIoU), establishing itself as the go-to general-purpose vision backbone.

The Problem: ViT Does Not Scale to Dense Vision Tasks

The original Vision Transformer (ViT) computes self-attention globally across all image patches. For an image tokenized into n patches, the attention computation is O(n^2) in both time and memory. This is manageable for classification at 224×224 (196 patches), but dense prediction tasks like object detection and semantic segmentation require high-resolution inputs (e.g., 1024×1024, yielding 4096 patches with 16×16 patch size). At that scale, global self-attention becomes prohibitively expensive.

Beyond computational cost, ViT has a structural limitation: it produces single-scale feature maps. CNNs naturally produce hierarchical, multi-scale features through pooling and strided convolutions — a property that feature pyramid networks (FPN), anchor-based detectors, and segmentation decoders all depend on. ViT's flat sequence of same-resolution tokens cannot directly serve these downstream architectures.

Swin Transformer addresses both problems simultaneously.

Window-Based Self-Attention

The core mechanism is simple: instead of computing attention across all n tokens, partition the feature map into non-overlapping local windows of fixed size M \times M (default M = 7) and compute self-attention independently within each window.

For a feature map of h \times w tokens, the computational complexity changes from:

to:

where C is the embedding dimension. The critical difference is in the second term: (hw)^2 becomes M^2 hw. Since M is fixed, the attention cost scales linearly with image size rather than quadratically. For a 1024×1024 image, Stage 1 has 256×256 tokens from 4×4 patches. The ratio of global to window attention terms is approximately 65536/49 \approx 1337 before boundary padding; this is not a whole-model speedup. Both token count and fixed channel width matter in the comparison.

The trade-off is that each window attends only to its local M^2 tokens, losing global receptive field. This is where the shifted window mechanism becomes essential.

Shifted Window Attention: Cross-Window Communication

Window-based attention in isolation creates hard boundaries between windows — tokens at the edge of one window cannot attend to adjacent tokens in the neighboring window. Swin Transformer solves this by alternating between two windowing configurations across consecutive transformer blocks.

In layer \ell, the feature map is partitioned with standard non-overlapping windows. In layer \ell + 1, the window grid is shifted by (\lfloor M/2 \rfloor, \lfloor M/2 \rfloor) token positions, so that interior shifted windows straddle the boundaries of four windows from the previous layer. This creates cross-window connections without any additional attention computation.

Formally, consecutive Swin Transformer blocks compute:

where W-MSA is standard window multi-head self-attention and SW-MSA is shifted window multi-head self-attention. As in a standard transformer block, each sub-layer carries a residual connection and layer normalization — the additive terms and LN in the equations above.

A naive implementation of shifted windows would increase the number of windows (some partial at the borders), creating an irregular computation pattern. The paper introduces an efficient cyclic shift approach: shift the feature map, apply standard windowing, then mask out attention between tokens that are not actually adjacent in the original layout. This keeps the number of windows constant and enables batched computation.

Relative Position Bias

Unlike ViT, which uses absolute positional embeddings, Swin Transformer injects positional information through a relative position bias added to each attention head:

where B \in \mathbb{R}^{M^2 \times M^2} is the relative position bias matrix. Since relative positions along each axis range from -M+1 to M-1, the bias is parameterized as a smaller matrix \hat{B} \in \mathbb{R}^{(2M-1) \times (2M-1)} that is indexed for each query-key pair. This adds negligible parameters but provides consistent improvements — the ablation study shows +1.2% top-1 accuracy on ImageNet compared to no positional encoding, and +0.8% compared to absolute position embeddings (Table 4).

Hierarchical Feature Maps via Patch Merging

Swin Transformer produces multi-scale feature maps by progressively reducing spatial resolution through patch merging layers, analogous to pooling in CNNs.

The architecture has four stages. Starting from an input image of H \times W \times 3:

• Stage 1: A patch partition layer splits the image into non-overlapping 4 \times 4 patches (each flattened to a 48-dim vector), followed by a linear embedding to C dimensions. Resolution: \frac{H}{4} \times \frac{W}{4}.
• Stage 2: A patch merging layer concatenates features of each 2 \times 2 group of neighboring patches (yielding 4C-dim vectors), then projects to 2C via a linear layer. Resolution: \frac{H}{8} \times \frac{W}{8}.
• Stage 3: Same merging operation, resolution \frac{H}{16} \times \frac{W}{16}, channel dimension 4C.
• Stage 4: Resolution \frac{H}{32} \times \frac{W}{32}, channel dimension 8C.

This produces a feature pyramid at 4x, 8x, 16x, and 32x downsampling — the same set of scales that FPN and other multi-scale architectures expect. This structural compatibility is what makes Swin Transformer a drop-in replacement for CNN backbones in detection and segmentation frameworks.

Architecture Variants

The paper defines four model sizes, all sharing the same architecture but varying in channel dimension and block depth:

• Swin-T (Tiny): C = 96, blocks = [2, 2, 6, 2], 29M params, 4.5 GFLOPs
• Swin-S (Small): C = 96, blocks = [2, 2, 18, 2], 50M params, 8.7 GFLOPs
• Swin-B (Base): C = 128, blocks = [2, 2, 18, 2], 88M params, 15.4 GFLOPs
• Swin-L (Large): C = 192, blocks = [2, 2, 18, 2], 197M params, 34.5 GFLOPs

The depth is concentrated in Stage 3 (18 blocks for S/B/L), which operates at 16x downsampling — a design choice that mirrors ResNet's heavy middle stages and balances computational cost against representational capacity.

Implementation: From Shifted Windows to Classification

The hierarchy now explains where features live and how neighboring windows communicate. The implementation below follows those operations in the same order, then separates a reduced offline training check from native pretrained classification. That distinction matters when comparing executable examples with the paper’s evaluation protocols.

Follow V1 from pixels to a classifier with eight executable rows. Original Microsoft source is pinned; rectangular and odd-grid mechanics explicitly use the later dynamic TorchVision port. The offline program needs no network or checkpoint. Genuine photo outputs below come from a separate native pretrained classifier.

Download mechanics PythonVerified requirementsRun commands and provenance

1. Embed each non-overlapping RGB patch

Original paper

Enlarge figure

Math and tensor shapes

X=\operatorname{LN}(\operatorname{Conv}_{4,4}(I))

X\in\mathbb R^{B\times(H/4)\times(W/4)\times C}

Kernel and stride four implement the same local linear projection as flattening the 48 pixels. The program uses channels-last features after convolution and requires image axes divisible by four. It keeps patch normalization; it adds no absolute positional embedding.

Smoke [2,3,64,64] → [2,16,16,24]; native [1,56,56,96]

Pinned Microsoft V1 implementation

Python implementation

python

class PatchEmbed(nn.Module):
    """Project non-overlapping 4x4 RGB patches, then normalize channels."""

    def __init__(self, dim):
        super().__init__()
        if dim <= 0:
            raise ValueError("Embedding width must be positive")
        self.projection = nn.Conv2d(3, dim, kernel_size=4, stride=4)
        self.norm = nn.LayerNorm(dim)

    def forward(self, images):
        if images.ndim != 4 or images.shape[1] != 3:
            raise ValueError("Expected [B,3,H,W] RGB images")
        if min(images.shape) <= 0 or any(n % 4 for n in images.shape[-2:]):
            raise ValueError("Image axes must be positive multiples of four")
        return self.norm(self.projection(images).permute(0, 2, 3, 1))

2. Partition the grid and retain an exact inverse

Original paper

Enlarge figure

Math and tensor shapes

n_W=hw/M^2

X_W\in\mathbb R^{(Bn_W)\times M^2\times C}

Reshape separates the window grid from coordinates inside each window; permutation brings those windows together. The reverse is an exact inverse, independently tested with numbered pixels and multiple batches. The regular native first stage has 64 windows per image.

Smoke [2,16,16,24] → [32,16,24]; native M=7 gives 49 tokens/window

Pinned Microsoft V1 implementation

Python implementation

python

def window_partition(x, size):
    """Reorder [B,H,W,C] into [B*nW,M*M,C] windows without mixing pixels."""
    validate_grid(x)
    b, h, w, c = x.shape
    if size <= 0 or h % size or w % size:
        raise ValueError("Window size must divide both grid axes")
    grouped = x.reshape(b, h // size, size, w // size, size, c)
    return grouped.permute(0, 1, 3, 2, 4, 5).reshape(-1, size * size, c)


def window_reverse(windows, size, height, width):
    """Invert partition exactly for a known divisible spatial grid."""
    if size <= 0 or min(height, width) <= 0 or height % size or width % size:
        raise ValueError("Reverse axes must be positive multiples of window size")
    count = (height // size) * (width // size)
    if windows.ndim != 3 or windows.shape[1] != size * size:
        raise ValueError("Expected [B*nW,M*M,C] windows")
    if min(windows.shape) <= 0 or windows.shape[0] % count:
        raise ValueError("Window count cannot reconstruct this grid")
    grouped = windows.reshape(
        -1, height // size, width // size, size, size, windows.shape[-1]
    )
    return grouped.permute(0, 1, 3, 2, 4, 5).reshape(
        -1, height, width, windows.shape[-1]
    )

3. Index directional relative offsets for every head

Original paper

Enlarge figure

Math and tensor shapes

r=\Delta y+M-1,\quad c=\Delta x+M-1

i_{qk}=r(2M-1)+c

B_a(q,k)=\hat B[i_{qk},a]

Reversing a pair complements the flattened index around (2M−1)²−1. That indexing relation does not force learned biases for opposite directions to be equal. These parameters are separate for each block and head.

Native index [49,49], table [169,heads], gathered bias [heads,49,49]

Pinned Microsoft V1 implementation

Python implementation

python

def relative_position_index(size):
    """Map signed query-minus-key offsets to (2M-1)^2 table entries."""
    if size <= 0:
        raise ValueError("Window size must be positive")
    coordinates = torch.stack(
        torch.meshgrid(torch.arange(size), torch.arange(size), indexing="ij")
    )
    coordinates = coordinates.flatten(1)
    offsets = coordinates[:, :, None] - coordinates[:, None, :]
    offsets = offsets.permute(1, 2, 0) + size - 1
    return offsets[..., 0] * (2 * size - 1) + offsets[..., 1]


def relative_bias(table, index):
    """Gather an independent learned bias for each head and ordered pair."""
    n = index.shape[0]
    return table[index.reshape(-1)].reshape(n, n, -1).permute(2, 0, 1)

4. Shift cyclically and mark disconnected edge regions

Original paper

Enlarge figure

Math and tensor shapes

\tilde X=\operatorname{roll}(\operatorname{pad}(X),-s)

A_{qk}=\begin{cases}0&r_q=r_k\\-100&r_q\ne r_k\end{cases}

Dynamic padding here follows the later TorchVision V1 port: pad bottom/right to a window multiple and disable each shifted axis independently if its padded extent fits one window. Microsoft’s fixed-resolution block instead reduces its window and disables both shifts when its smallest extent is at most M. The finite mask makes wrapped attention negligible for ordinary logits; it does not guarantee mathematical zero. Padding is not a separate validity mask in this port.

Smoke s=(2,2), M=4; mask [nW,16,16]

Pinned Microsoft V1 implementation

Later TorchVision dynamic port

Python implementation

python

def shifted_grid(x, size, shift):
    """Pad then disable a shift independently along each single-window axis."""
    validate_grid(x)
    if size <= 0 or not 0 <= shift < size:
        raise ValueError("Require 0 <= shift < positive window size")
    height, width = x.shape[1:3]
    x = F.pad(x, (0, 0, 0, (-width) % size, 0, (-height) % size))
    sy = shift if x.shape[1] > size else 0
    sx = shift if x.shape[2] > size else 0
    return torch.roll(x, shifts=(-sy, -sx), dims=(1, 2)), (sy, sx)


def region_mask(height, width, size, shifts, device=None, dtype=torch.float32):
    """Use finite -100 between cyclic regions, zero within each region."""
    sy, sx = shifts
    regions = torch.zeros((1, height, width, 1), device=device)
    ys = (
        (slice(0, -size), slice(-size, -sy), slice(-sy, None)) if sy else (slice(None),)
    )
    xs = (
        (slice(0, -size), slice(-size, -sx), slice(-sx, None)) if sx else (slice(None),)
    )
    label = 0
    for y in ys:
        for x in xs:
            regions[:, y, x, :] = label
            label += 1
    labels = window_partition(regions, size).squeeze(-1)
    different = labels[:, :, None] != labels[:, None, :]
    return different.to(dtype) * -100.0

5. Apply attention, reverse windows, then undo the shift

Original paper

Enlarge figure

Math and tensor shapes

S=d^{-1/2}QK^T+B+A

P=\operatorname{softmax}(S)

T=\operatorname{reverse}(PVW_O)

Y=\operatorname{crop}(\operatorname{roll}(T,s))

The program projects Q/K/V with bias, gathers per-head relative bias and broadcasts the region mask over batches and heads. It restores windows, rolls by the positive shift and crops the dynamic padding. This is V1 scaled dot-product attention, not V2 cosine attention or continuous relative bias.

Native Q,K,V [B*nW,heads,49,32] → original [B,h,w,C]

Pinned Microsoft V1 implementation

Later TorchVision dynamic port

Python implementation

python

def forward(self, x):
    height, width = x.shape[1:3]
    shifted, shifts = shifted_grid(x, self.size, self.shift)
    ph, pw = shifted.shape[1:3]
    windows = window_partition(shifted, self.size)
    batch_windows, n, c = windows.shape
    qkv = self.qkv(windows).reshape(
        batch_windows, n, 3, self.heads, c // self.heads
    )
    q, k, v = qkv.permute(2, 0, 3, 1, 4).unbind(0)
    logits = (q * ((c // self.heads) ** -0.5)) @ k.transpose(-2, -1)
    logits = logits + relative_bias(self.bias_table, self.index)
    mask = region_mask(ph, pw, self.size, shifts, x.device, logits.dtype)
    nw = mask.shape[0]
    logits = logits.reshape(-1, nw, self.heads, n, n) + mask[None, :, None]
    weights = logits.reshape(batch_windows, self.heads, n, n).softmax(-1)
    attended = (weights @ v).transpose(1, 2).reshape(batch_windows, n, c)
    restored = window_reverse(self.projection(attended), self.size, ph, pw)
    restored = torch.roll(restored, shifts=shifts, dims=(1, 2))
    return restored[:, :height, :width, :].contiguous()

6. Keep both V1 residual branches pre-normalized

Original paper

Enlarge figure

Math and tensor shapes

R=\operatorname{LN}(X)

U=X+\operatorname{(S)WMSA}(R)

V=\operatorname{LN}(U)

Y=U+W_2\operatorname{GELU}(W_1V)

Consecutive blocks alternate regular and shifted windows. The mechanics profile disables dropout and stochastic depth so the CPU behavior is deterministic; the paper’s training recipe includes stochastic depth. Moving normalization after the residual would change V1 into a different block.

[B,h,w,C] retained; MLP expands C → 4C → C

Pinned Microsoft V1 implementation

Python implementation

python

class SwinBlock(nn.Module):
    """Pre-normalized V1 attention/residual and GELU MLP/residual."""

    def __init__(self, dim, heads, size=4, shift=0):
        super().__init__()
        self.norm1 = nn.LayerNorm(dim)
        self.attention = WindowAttention(dim, heads, size, shift)
        self.norm2 = nn.LayerNorm(dim)
        self.mlp = nn.Sequential(
            nn.Linear(dim, 4 * dim), nn.GELU(), nn.Linear(4 * dim, dim)
        )

    def forward(self, x):
        x = x + self.attention(self.norm1(x))
        return x + self.mlp(self.norm2(x))

7. Merge four neighbors before doubling channel width

Original paper

Enlarge figure

Math and tensor shapes

v=[x_{00};x_{10};x_{01};x_{11}]

Y=\operatorname{LN}(v)W_{4C\to2C}

Neighbor order is top-left, bottom-left, top-right, bottom-right, matching release channel order. Original PatchMerging asserts even H/W. The dynamic port used here pads an odd bottom/right edge with zeros and produces ceil(H/2)×ceil(W/2). V1 normalizes before reduction; V2 reverses that order.

Smoke [2,16,16,24] → [2,8,8,48]; native 56→28→14→7

Pinned Microsoft V1 implementation

Later TorchVision dynamic port

Python implementation

python

def merge_neighbors(x):
    """Concatenate x00,x10,x01,x11; dynamic port pads odd spatial axes."""
    validate_grid(x)
    height, width = x.shape[1:3]
    x = F.pad(x, (0, 0, 0, width % 2, 0, height % 2))
    return torch.cat(
        (x[:, 0::2, 0::2], x[:, 1::2, 0::2], x[:, 0::2, 1::2], x[:, 1::2, 1::2]), dim=-1
    )


class PatchMerging(nn.Module):
    """Normalize the four-neighbor vector before its 4C-to-2C projection."""

    def __init__(self, dim):
        super().__init__()
        if dim <= 0:
            raise ValueError("Channel width must be positive")
        self.norm = nn.LayerNorm(4 * dim)
        self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)

    def forward(self, x):
        return self.reduction(self.norm(merge_neighbors(x)))

8. Build the hierarchy and classify its pooled features

Original paper

Enlarge figure

Math and tensor shapes

g=(hw)^{-1}\sum_{i,j}\operatorname{LN}(X_{ij})

z=gW+b,\quad L=CE(z,y)

The offline smoke instead uses width 24, depths [2,2,2,2], windows 4, 64×64 random inputs and five classes. One AdamW update reaches the whole hierarchy. It omits ImageNet data, augmentation, the 300-epoch schedule and trained semantics. The separate genuine exporter runs native TorchVision Swin-T V1 with strict released weights and 1000 classes.

Native [1,7,7,768] → [1,768] → [1,1000]

Pinned Microsoft V1 implementation

Python implementation

python

class SwinMechanics(nn.Module):
    """Four reduced stages, alternating regular and shifted blocks."""

    def __init__(
        self, dim=24, depths=(2, 2, 2, 2), heads=(3, 6, 12, 24), size=4, classes=5
    ):
        super().__init__()
        if len(depths) != 4 or len(heads) != 4 or min(depths) <= 0 or classes <= 0:
            raise ValueError("Require four nonempty stages and positive classes")
        self.patch_embed = PatchEmbed(dim)
        self.stages, self.merges = nn.ModuleList(), nn.ModuleList()
        for stage, depth in enumerate(depths):
            width = dim * 2**stage
            self.stages.append(
                nn.Sequential(
                    *[
                        SwinBlock(
                            width,
                            heads[stage],
                            size,
                            0 if block % 2 == 0 else size // 2,
                        )
                        for block in range(depth)
                    ]
                )
            )
            if stage < 3:
                self.merges.append(PatchMerging(width))
        self.norm = nn.LayerNorm(8 * dim)
        self.head = nn.Linear(8 * dim, classes)

    def forward(self, images):
        x = self.patch_embed(images)
        shapes = []
        for stage, blocks in enumerate(self.stages):
            x = blocks(x)
            shapes.append(list(x.shape))
            if stage < 3:
                x = self.merges[stage](x)
        logits = self.head(self.norm(x).mean(dim=(1, 2)))
        return logits, shapes


def train_step(model, optimizer, images, labels):
    """Update the entire random-weight hierarchy with supervised cross entropy."""
    logits, shapes = model(images)
    if labels.shape != (images.shape[0],) or labels.dtype != torch.long:
        raise ValueError("Require one integer class label per image")
    if labels.numel() == 0 or labels.min() < 0 or labels.max() >= logits.shape[1]:
        raise ValueError("Class labels are outside the classifier vocabulary")
    optimizer.zero_grad(set_to_none=True)
    loss = F.cross_entropy(logits, labels)
    loss.backward()
    optimizer.step()
    return loss.detach(), shapes

Inspect genuine Swin-T V1 classification

These two CC0 photos were processed by native TorchVision Swin-T with Swin_T_Weights.IMAGENET1K_V1. Its published ImageNet-1K top-1 is 81.474% under the TorchVision recipe; original paper Swin-T reports 81.3%. This export is two inferences, with no benchmark evaluation. Softmax scores describe an uncalibrated 1000-class classifier.

Exact preprocessing: bicubic shorter side 232, center crop 224×224, RGB float pixels divided by 255, then ImageNet mean [0.485,0.456,0.406] and standard deviation [0.229,0.224,0.225]. The pictured crops are the actual processed inputs; normalized inputs and all 1000 logits/probabilities are saved.

Chelsea the cat

Top five of 1000 classes

Class	Logit	Softmax
Egyptian cat	8.731	72.88%
tabby	6.468	7.58%
tiger cat	6.394	7.04%
lynx	3.552	0.41%
Persian cat	2.071	0.09%

Exact normalized input and full outputs (.npz)

Photo: Stefan van der Walt · source · CC0 license

Cup of coffee

Top five of 1000 classes

Class	Logit	Softmax
espresso	9.621	91.31%
cup	5.553	1.56%
coffee mug	3.860	0.29%
consomme	3.419	0.19%
chocolate sauce	3.264	0.16%

Exact normalized input and full outputs (.npz)

Photo: Rachel Michetti · source · CC0 license

All logits, probabilities and hashesNative exporter PythonPhoto provenancePinned original source hashesOriginal source MIT license

Key Results

ImageNet-1K Classification: With ImageNet-1K training, Swin-T achieves 81.3% top-1 at 224×224. Table 1(a) reports Swin-B at 83.5% for 224×224 and 84.5% for 384×384, versus DeiT-B’s 81.8% and 83.1% at those respective resolutions. The §4.1 prose instead gives 83.3% for Swin-B at 224×224, so the paper’s table and prose disagree on that value. With ImageNet-22K pretraining followed by ImageNet-1K fine-tuning, Swin-L reaches 87.3% at 384×384. These are the original single-crop classification results in Table 1 and §4.1, distinct from the later TorchVision checkpoint recipe above.

COCO Object Detection: The best system uses Swin-L with ImageNet-22K pretraining, an improved Hybrid Task Cascade (HTC++) detector and multi-scale testing: 58.7 box AP and 51.1 mask AP on COCO test-dev. These exceed Copy-paste by +2.7 box AP and DetectoRS by +2.6 mask AP. The separate comparable Cascade Mask R-CNN comparison is Swin-B at 51.9 box AP/45.0 mask AP versus ResNeXt-101-64x4d at 48.3/41.7. Keep these two protocols separate (Table 2 and §4.2).

ADE20K Semantic Segmentation: With UperNet and ImageNet-22K pretraining, Swin-L achieves 53.5 mIoU on ADE20K validation. Its +3.2 mIoU comparison is against SETR’s 50.3, not another UperNet model. The hierarchy provides the multi-scale features consumed by the decoder (Table 3 and §4.3).

Critical Analysis

Strengths:

• Linear complexity enables processing high-resolution inputs that are infeasible for global attention models. This is not just a theoretical advantage — it enabled practical deployment on tasks like detection and segmentation where 800×1333 inputs are standard.
• Multi-scale feature maps make Swin Transformer structurally compatible with the entire ecosystem of dense prediction heads (FPN, UperNet, Cascade R-CNN), eliminating the need for task-specific architectural redesign.
• General-purpose backbone: the same pretrained model serves classification, detection, segmentation, and later video understanding, reducing engineering overhead compared to task-specific architectures.

Limitations:

• Window boundary artifacts: despite shifted windows, the effective receptive field grows slowly across layers. Tasks requiring long-range spatial reasoning (e.g., counting distant objects, understanding scene layout) may suffer compared to global attention.
• Implementation complexity: the cyclic shift with attention masking is considerably more involved than standard transformer attention. This complicates custom kernel development and makes the model harder to port to new hardware.
• Fixed window size: the M = 7 window size is a hyperparameter that does not adapt to content. Regions with large homogeneous textures waste capacity on uninformative local attention, while regions with fine detail may need more than 7×7 context.
• Training recipe sensitivity: achieving the reported numbers requires careful use of augmentation (RandAugment, Mixup, CutMix), regularization (stochastic depth, label smoothing), and the AdamW optimizer with cosine learning rate decay. The architecture alone, without this recipe, underperforms.

Impact and Legacy

Swin Transformer became the dominant vision backbone for the 2021–2023 period, displacing both CNN-based (ResNet, EfficientNet) and earlier ViT-based approaches in detection and segmentation benchmarks. Its influence extended well beyond classification:

Swin V2 (2022) extended the architecture to 3 billion parameters and 1536×1536 resolution by replacing the learned bias matrix with a log-spaced continuous relative position bias, addressing the resolution transfer problem. Video Swin Transformer applied the shifted window mechanism to 3D spatiotemporal volumes for video understanding. The architecture also became the default backbone for medical imaging (Swin UNETR for 3D segmentation) and remote sensing.

The broader legacy is architectural: Swin Transformer showed that locality and hierarchy are useful inductive biases for vision transformers alongside data and training choices. Its windows are non-overlapping and shifted between blocks, rather than sliding around every query; the boundaries also mean exact translation equivariance is not guaranteed. This direction influenced subsequent architectures like ConvNeXt, which showed CNNs could match Swin when given the same training recipe, and MaxViT, which combined local and global attention.

Related Reading

• Attention Is All You Need — the original transformer architecture that Swin adapts for vision
• Vision Transformer (ViT) — the baseline global-attention ViT that Swin improves upon
• DETR — another approach to transformers in detection, using global attention with learned object queries
• EfficientNet — the CNN scaling methodology that Swin displaced as the top ImageNet backbone
• BEiT — masked image modeling pretraining that further improved Swin Transformer performance
• Hilbert-Guided Sparse Local Attention — reorders tokens along a Hilbert curve so window attention runs fast on block-sparse kernels
• Deep Residual Learning (ResNet) — the hierarchical CNN backbone design that Swin Transformer's feature pyramid mirrors