Plain ViT Backbones for Object Detection

TL;DR

Object detection has historically relied on hierarchical backbones (ResNet, Swin Transformer) that produce multi-scale feature pyramids. This paper asks whether a plain, non-hierarchical ViT, with single-scale features, can serve as a competitive detection backbone. ViTDet independently rescales the one final ViT map into pyramid levels and uses window attention with a few global propagation blocks during fine-tuning. In paper v2, Table 6, the MAE-pretrained ViT-H with Cascade Mask R-CNN reaches 60.4 box AP on COCO with single-scale testing; ViT-L reaches the same number with multi-scale testing. These are trained system results, not a property of a feature-pyramid implementation alone.

The Core Challenge: Single-Scale vs. Multi-Scale

The dominant paradigm in object detection before this paper was to use hierarchical backbones that naturally produce multi-scale feature pyramids. CNN-based detectors rely on feature pyramid networks (FPN) that extract features at multiple spatial resolutions. A ResNet, for instance, naturally produces feature maps at strides of 4, 8, 16, and 32 pixels through its successive pooling and strided convolution layers. Hierarchical vision transformers like Swin and MViTv2 were designed specifically to replicate this multi-scale structure, introducing progressively reduced spatial resolution at each stage.

These multi-scale features are critical for detecting objects across a wide range of sizes — small objects are best detected from high-resolution, low-stride feature maps, while large objects benefit from semantically richer, low-resolution maps.

A plain ViT has none of this structure. It processes an image as a flat sequence of non-overlapping patches (typically 16 \times 16 pixels), applies L transformer blocks of identical dimension, and produces a single-scale feature map at stride 16. There is no downsampling, no resolution hierarchy, and no natural place to tap multi-scale features.

The computational structure of these two approaches differs fundamentally. In a hierarchical backbone, the feature map spatial dimensions shrink at each stage while channel dimensions grow, maintaining roughly constant FLOPs per stage. The total computation distributes across resolution levels:

where C_s is the channel dimension and r_s is the spatial reduction factor at stage s. In a plain ViT, all L layers operate at the same resolution with the same hidden dimension d, making the cost uniform across depth:

where N = HW / p^2 is the number of patches. The question is whether this architectural simplicity is a fundamental limitation or merely a matter of adaptation.

Method: Minimal Adaptations to Plain ViT

The paper preserves the plain backbone's single spatial resolution. During detection fine-tuning it adapts attention for larger inputs and adds a simple pyramid between the final backbone map and the detector.

Simple Feature Pyramid from the Final Map. Section 3 and Figure 2c use only the last stride-16 feature map. Four parallel branches rescale that same tensor. The appendix and official release specify:

• Stride 4: two successive 2 \times transposed convolutions, with channel LayerNorm and GELU between them
• Stride 8: one 2 \times deconvolution
• Stride 16: identity (the native ViT resolution)
• Stride 32: stride-2 max pooling

Each branch then applies a 1×1 convolution with LayerNorm to obtain 256 channels, followed by a 3×3 convolution with LayerNorm. There are no lateral or top-down connections. Intermediate-depth taps belong to the compared FPN-like alternative in Figure 2a, not the selected simple pyramid.

Window Attention for High-Resolution Inputs. Standard global self-attention has quadratic pairwise cost. A 1024×1024 image with patch size 16 gives 4096 tokens and a 4096×4096 attention matrix per head. Fine-tuning instead uses regular, unshifted attention windows with side length s = 14 tokens, hence m = s² = 196 tokens per window. The pairwise cost is O(Ns^2)=O(Nm), rather than O(N^2). For a non-divisible grid, replace N by the padded token count. The official release pads normalized tokens with zeros, includes padded positions in attention without a validity mask, then crops back to the original geometry.

The critical design choice is how to maintain cross-window information flow. Without it, each window operates in isolation and the model cannot reason about objects that span multiple windows. The authors address this by designating a small number of global attention blocks (typically 4 evenly spaced blocks out of 24 total) that compute full N \times N attention. These global blocks act as information bridges — features from different spatial regions can interact at these layers, while the remaining windowed layers handle local processing efficiently. This is a simpler approach than Swin Transformer’s shifted window mechanism, which achieves cross-window communication through alternating window configurations at every layer.

Pre-training Compatibility. A plain ViT can reuse existing MAE (Masked Autoencoder) pretraining without redesigning its pretraining architecture. Detection fine-tuning adapts attention and absolute positions for larger grids, adds relative position terms in the comparisons, and trains the pyramid and heads. The appendix notes a further ViT-H detail: its pretrained 14×14 patch filters are interpolated to 16×16. Compatibility does not mean every checkpoint tensor transfers without adaptation.

Why a Pyramid from One Final Map Works

The pyramid does not need to imitate a hierarchy inside the backbone. In Table 1, the simple final-map pyramid matches or exceeds both FPN alternatives: one with four depth taps and one using only the last map with FPN connections. For the MAE-pretrained ViT-L with Mask R-CNN, the simple pyramid achieves 54.6 box AP versus 51.2 without a pyramid. The evidence supports building multiple output scales, without establishing a need for intermediate-depth features.

Here F_L is the single final stride-16 map and g_r is an independent rescaling and per-level projection for r \in \{4, 8, 16, 32\}. This is authored notation for Figure 2c. Every level shares the same source tensor, but has its own branch parameters. The detector maps anchors and regions to pyramid levels according to object scale, as in FPN.

Window attention controls the cost of producing F_L, while the selected global blocks permit information to cross window boundaries. The backbone still keeps one resolution throughout its depth. The pyramid introduces multiple resolutions only after the last block, handing stride-labelled maps to the RPN and RoI heads of Mask R-CNN or Cascade Mask R-CNN.

That separation is the useful implementation boundary: execute patches, local/global attention, position adaptation and all four pyramid branches, then stop before proposal generation, RoI processing, and detection losses. The reference below makes this boundary explicit; the following results describe complete trained detectors.

Implementation

class PatchGrid(nn.Module):
    """Project nonoverlapping 16-pixel patches into a BHWC grid.

    Inputs need at least one full patch; incomplete right/bottom pixels are
    discarded by the unpadded convolution, as in the release. Detection inputs
    normally arrive padded to stride 32 upstream; this example does not invent
    image padding inside the patch projector.
    """

    def __init__(self, channels=8):
        super().__init__()
        if not positive_int(channels):
            raise ValueError("channels must be positive")
        self.proj = nn.Conv2d(3, channels, kernel_size=16, stride=16)

    def forward(self, image):
        """Return [B, floor(H/16), floor(W/16), C] patch embeddings."""
        validate_grid(image)
        if image.shape[1] != 3 or min(image.shape[-2:]) < 16:
            raise ValueError(
                "Expected RGB images with at least one 16x16 patch"
            )
        return self.proj(image).permute(0, 2, 3, 1)


def get_abs_pos(positions, has_cls_token, hw):
    """Remove the pretraining CLS position and bicubic-resize a square grid."""
    if len(hw) != 2 or not all(positive_int(size) for size in hw):
        raise ValueError("Target height and width must be positive integers")
    if positions.ndim != 3 or positions.shape[0] != 1:
        raise ValueError("Absolute positions must have shape [1, N, C]")
    spatial = positions[:, 1:] if has_cls_token else positions
    side = math.isqrt(spatial.shape[1])
    if side < 1 or side * side != spatial.shape[1] or spatial.shape[2] < 1:
        raise ValueError("Pretraining spatial positions must form a square")
    grid = spatial.reshape(1, side, side, -1)
    if (side, side) == tuple(hw):
        return grid
    resized = F.interpolate(
        grid.permute(0, 3, 1, 2),
        size=hw,
        mode="bicubic",
        align_corners=False,
    )
    return resized.permute(0, 2, 3, 1)


def feature_map(tokens):
    """Expose only spatial tokens as an NCHW detector-interface map."""
    validate_grid(tokens)
    return tokens.permute(0, 3, 1, 2)

def window_partition(x, window_side=14):
    """Pad BHWC at bottom/right with zeros and return row-major windows.

    Returns:
        Windows [B*ceil(H/s)*ceil(W/s), s, s, C] and padded (Hp, Wp).
        No validity mask is returned: padded tokens enter ordinary attention.
    """
    validate_grid(x)
    if not positive_int(window_side):
        raise ValueError("window_side must be a positive integer")
    b, h, w, c = x.shape
    pad_h, pad_w = (-h) % window_side, (-w) % window_side
    padded = F.pad(x, (0, 0, 0, pad_w, 0, pad_h))
    hp, wp = h + pad_h, w + pad_w
    tiles = padded.reshape(
        b,
        hp // window_side,
        window_side,
        wp // window_side,
        window_side,
        c,
    )
    windows = tiles.permute(0, 1, 3, 2, 4, 5).reshape(
        -1, window_side, window_side, c
    )
    return windows, (hp, wp)

def get_rel_pos(q_size, k_size, table):
    """Linearly resize a relative-position table and gather scaled offsets."""
    if not positive_int(q_size) or not positive_int(k_size):
        raise ValueError("Query/key sizes must be positive integers")
    if table.ndim != 2 or min(table.shape) < 1:
        raise ValueError("Relative positions must have shape [L, head_dim]")
    length = 2 * max(q_size, k_size) - 1
    if table.shape[0] != length:
        table = F.interpolate(
            table.T.unsqueeze(0),
            size=length,
            mode="linear",
            align_corners=False,
        )[0].T
    q = torch.arange(q_size, device=table.device)[:, None]
    k = torch.arange(k_size, device=table.device)[None, :]
    offsets = q * max(k_size / q_size, 1.0)
    offsets = offsets - k * max(q_size / k_size, 1.0)
    offsets = offsets + (k_size - 1) * max(q_size / k_size, 1.0)
    return table[offsets.long()]


def add_rel_pos(scores, q, rel_h, rel_w, hw):
    """Add query-dependent decomposed height/width terms to self-attention."""
    h, w = hw
    rh, rw = get_rel_pos(h, h, rel_h), get_rel_pos(w, w, rel_w)
    grid = q.reshape(q.shape[0], h, w, -1)
    height = torch.einsum("bhwc,hkc->bhwk", grid, rh)
    width = torch.einsum("bhwc,wkc->bhwk", grid, rw)
    expanded = scores.reshape(q.shape[0], h, w, h, w)
    return (expanded + height[..., None] + width[..., None, :]).reshape_as(
        scores
    )


class Attention(nn.Module):
    """Ordinary multi-head spatial attention, including padded tokens."""

    def __init__(
        self, channels=8, heads=2, use_rel_pos=False, relative_grid=(14, 14)
    ):
        super().__init__()
        if (
            not positive_int(channels)
            or not positive_int(heads)
            or channels % heads
        ):
            raise ValueError("Positive channels must be divisible by heads")
        if len(relative_grid) != 2 or not all(
            positive_int(s) for s in relative_grid
        ):
            raise ValueError("relative_grid must contain two positive sizes")
        self.channels, self.heads = channels, heads
        self.scale = (channels // heads) ** -0.5
        self.qkv = nn.Linear(channels, 3 * channels, bias=True)
        self.proj = nn.Linear(channels, channels, bias=True)
        self.use_rel_pos = use_rel_pos
        if use_rel_pos:
            self.rel_h = nn.Parameter(
                torch.zeros(2 * relative_grid[0] - 1, channels // heads)
            )
            self.rel_w = nn.Parameter(
                torch.zeros(2 * relative_grid[1] - 1, channels // heads)
            )

    def forward(self, x):
        """Return BHWC with scaled QK scores and optional relative positions."""
        validate_grid(x)
        b, h, w, c = x.shape
        if c != self.channels:
            raise ValueError(
                "Input channels disagree with attention projection"
            )
        qkv = self.qkv(x).reshape(b, h * w, 3, self.heads, c // self.heads)
        q, k, v = (
            qkv.permute(2, 0, 3, 1, 4)
            .reshape(3, b * self.heads, h * w, c // self.heads)
            .unbind(0)
        )
        scores = (q * self.scale) @ k.transpose(-2, -1)
        if self.use_rel_pos:
            scores = add_rel_pos(scores, q, self.rel_h, self.rel_w, (h, w))
        values = scores.softmax(-1) @ v
        merged = values.reshape(b, self.heads, h, w, -1)
        merged = merged.permute(0, 2, 3, 1, 4).reshape(b, h, w, c)
        return self.proj(merged)

def window_unpartition(windows, window_side, padded_hw, original_hw):
    """Restore row-major BHWC windows and crop bottom/right padding exactly."""
    validate_grid(windows)
    if not positive_int(window_side):
        raise ValueError("window_side must be a positive integer")
    if len(padded_hw) != 2 or len(original_hw) != 2:
        raise ValueError("Geometry must contain height and width")
    hp, wp = padded_hw
    h, w = original_hw
    if not all(positive_int(s) for s in (hp, wp, h, w)):
        raise ValueError("Geometry sizes must be positive integers")
    if hp != h + (-h) % window_side or wp != w + (-w) % window_side:
        raise ValueError("Padded geometry must match minimal window padding")
    if tuple(windows.shape[1:3]) != (window_side, window_side):
        raise ValueError("Window tensor has the wrong tile size")
    per_image = (hp // window_side) * (wp // window_side)
    if windows.shape[0] % per_image:
        raise ValueError("Window count does not represent complete images")
    b, c = windows.shape[0] // per_image, windows.shape[-1]
    tiles = windows.reshape(
        b,
        hp // window_side,
        wp // window_side,
        window_side,
        window_side,
        c,
    )
    restored = tiles.permute(0, 1, 3, 2, 4, 5).reshape(b, hp, wp, c)
    return restored[:, :h, :w].contiguous()

def global_slots(depth, count=4):
    """Choose zero-based last blocks of equally sized depth subsets."""
    if not positive_int(depth) or not positive_int(count) or depth % count:
        raise ValueError("depth must be positive and divisible by global count")
    return tuple(range(depth // count - 1, depth, depth // count))


class Block(nn.Module):
    """Pre-LN attention then pre-LN MLP; window_side=0 selects global mixing."""

    def __init__(self, channels=8, heads=2, window_side=14, use_rel_pos=False):
        super().__init__()
        if (
            not isinstance(window_side, int)
            or isinstance(window_side, bool)
            or window_side < 0
        ):
            raise ValueError("window_side must be a nonnegative integer")
        self.window_side = window_side
        self.norm1 = nn.LayerNorm(channels, eps=1e-6)
        self.attn = Attention(channels, heads, use_rel_pos)
        self.norm2 = nn.LayerNorm(channels, eps=1e-6)
        self.mlp = nn.Sequential(
            nn.Linear(channels, 4 * channels),
            nn.GELU(),
            nn.Linear(4 * channels, channels),
        )

    def forward(self, x):
        """Pad only normalized spatial tokens; crop before shortcut addition."""
        normalized = self.norm1(x)
        if self.window_side:
            windows, padded = window_partition(normalized, self.window_side)
            attended = window_unpartition(
                self.attn(windows),
                self.window_side,
                padded,
                tuple(x.shape[1:3]),
            )
        else:
            attended = self.attn(normalized)
        x = x + attended
        return x + self.mlp(self.norm2(x))


class Backbone(nn.Module):
    """Reduced plain ViT with one final map, no depth taps or classifier token."""

    def __init__(self, channels=8, heads=2, depth=4, globals_count=2):
        super().__init__()
        self.global_indexes = global_slots(depth, globals_count)
        self.patch = PatchGrid(channels)
        self.positions = nn.Parameter(torch.empty(1, 197, channels))
        nn.init.trunc_normal_(self.positions, std=0.02)
        self.blocks = nn.ModuleList(
            [
                Block(
                    channels,
                    heads,
                    0 if index in self.global_indexes else 14,
                    use_rel_pos=True,
                )
                for index in range(depth)
            ]
        )
        for layer in self.modules():
            if isinstance(layer, nn.Linear):
                nn.init.trunc_normal_(layer.weight, std=0.02)
                nn.init.zeros_(layer.bias)

    def forward(self, image):
        """Return final [B, C, H/16, W/16] when image sides divide by 16."""
        x = self.patch(image)
        x = x + get_abs_pos(self.positions, True, tuple(x.shape[1:3]))
        for block in self.blocks:
            x = block(x)
        return feature_map(x)

class SimplePyramid(nn.Module):
    """Four independent release-style rescaling branches from ONE final map.

    Patch stride 16 yields p2/p3/p4/p5 at nominal strides 4/8/16/32.
    Output channels are reduced to 4, rather than the paper's 256.
    No lateral/top-down connections and no optional fifth top-block level.
    """

    strides = {"p2": 4, "p3": 8, "p4": 16, "p5": 32}

    def __init__(self, channels=8, output_channels=4):
        super().__init__()
        if (
            not positive_int(channels)
            or channels % 4
            or not positive_int(output_channels)
        ):
            raise ValueError(
                "Input width must divide by 4; output width must be positive"
            )
        self.channels = channels
        self.stages = nn.ModuleList()
        rescalers = [
            (
                [
                    nn.ConvTranspose2d(channels, channels // 2, 2, 2),
                    ChannelNorm(channels // 2),
                    nn.GELU(),
                    nn.ConvTranspose2d(channels // 2, channels // 4, 2, 2),
                ],
                channels // 4,
            ),
            (
                [nn.ConvTranspose2d(channels, channels // 2, 2, 2)],
                channels // 2,
            ),
            ([], channels),
            ([nn.MaxPool2d(2, 2)], channels),
        ]
        for rescale, width in rescalers:
            self.stages.append(
                nn.Sequential(
                    *rescale,
                    nn.Conv2d(width, output_channels, 1, bias=False),
                    ChannelNorm(output_channels),
                    nn.Conv2d(
                        output_channels,
                        output_channels,
                        3,
                        padding=1,
                        bias=False,
                    ),
                    ChannelNorm(output_channels),
                )
            )

    def forward(self, final_map):
        """Return the detector interface; odd sides use floor max-pool sizes."""
        validate_grid(final_map)
        if final_map.shape[1] != self.channels or min(final_map.shape[-2:]) < 2:
            raise ValueError("Expected correct channels and spatial sides >= 2")
        return {
            name: stage(final_map)
            for name, stage in zip(self.strides, self.stages)
        }

python -m pip install -r requirements-vitdet.txt
python vitdet.py --smoke --seed 7

Key Results

The experiments use Mask R-CNN and Cascade Mask R-CNN as detection frameworks on COCO.

Plain ViT scales competitively. Paper v2, Table 5 reports COCO box AP of 51.6/55.6/56.7 for MAE-pretrained ViT-B/L/H with Mask R-CNN, and 54.0/57.6/58.7 with Cascade Mask R-CNN. Table 6 is a separate system comparison using larger 1280-pixel inputs, relative position terms and soft-NMS: Cascade Mask R-CNN with ViT-H reaches 60.4 single-scale and 61.3 multi-scale box AP; ViT-L reaches 59.6 and 60.4 respectively. These recipes must not be mixed with Table 1's Mask R-CNN ablation.

MAE pre-training is critical. When initialized with MAE pre-training, ViT-L outperforms its supervised ImageNet-pretrained counterpart by approximately 4 AP. This gap is substantially larger than the pre-training improvement seen with hierarchical backbones (where MAE yields roughly 1–2 AP improvement), suggesting that plain ViTs benefit disproportionately from self-supervised pre-training for dense prediction tasks. The likely explanation is that MAE forces the model to learn strong local and spatial representations through the masking-and-reconstruction objective — exactly the kind of spatial reasoning that hierarchical architectures build in through their multi-scale structure but that a plain ViT must learn from data.

Simplicity has computational benefits. The plain ViT backbone, despite operating at a single scale internally, achieves competitive FLOPs and throughput compared to hierarchical alternatives. The regular, non-branching structure of plain ViT is also more hardware-friendly — it maps cleanly to modern accelerators without the irregular memory access patterns introduced by multi-scale processing.

Scaling behavior is strong. The paper demonstrates consistent improvements when scaling from ViT-B (86M parameters) to ViT-L (304M) to ViT-H (632M). The AP improvements are roughly log-linear with model size, and importantly, the gap between plain ViT and hierarchical backbones narrows with scale. At the ViT-H level, the plain backbone is unambiguously better than any hierarchical alternative tested, suggesting that the multi-scale inductive bias becomes less important as model capacity increases.

Window attention configuration matters. The ablation studies show that using 4 global attention blocks (out of 24 total) interspersed with windowed attention blocks provides a good balance. Using fewer global blocks degrades cross-region reasoning, while using more global blocks increases cost without proportionate accuracy gains. The window size of 14 \times 14 tokens was chosen to match the pre-training resolution of 224 / 16 = 14 patches per side, allowing positional embedding reuse.

Critical Analysis

Strengths:

• The paper makes a strong case for architectural simplicity. Rather than designing increasingly complex hierarchical vision transformers (Swin, PVT, MViTv2, each adding architectural innovations), it shows that a plain ViT with trivial adaptations can match these systems. This is a useful counterpoint to the trend of ever-more-complex architectures.
• The experimental methodology is thorough, with controlled comparisons across backbone sizes (ViT-B, L, H), pre-training strategies (supervised, MAE), and detection frameworks. The ablation studies on window attention configurations and feature pyramid design are well-structured.
• The practical implication is significant: teams can use a single plain ViT checkpoint for both classification and detection, simplifying the model development pipeline. This “one backbone, many heads” approach reduces engineering complexity and the computational cost of maintaining separate pre-trained models for different tasks.

Limitations:

• The windowed attention mechanism and feature pyramid construction, while simple, are not entirely “plain.” The claim of using a plain ViT backbone is slightly overstated — the detection-specific adaptations (deconvolutions for upsampling, windowed attention) add architectural complexity, even if it is lightweight.
• The strong results depend heavily on MAE pre-training. Without it, the plain ViT backbone underperforms hierarchical alternatives by a meaningful margin. This couples the detector’s effectiveness to the quality of self-supervised pre-training.
• The appendix also tests one-stage RetinaNet and observes similar scaling trends. It does not establish the same conclusion for FCOS, YOLO-style or DETR-family detectors.
• Inference latency analysis is limited. FLOPs and throughput are reported, but actual latency measurements across batch sizes and hardware configurations would strengthen the efficiency claims.
• The comparison with DETR-family detectors is notably absent. DETR and Deformable DETR take a fundamentally different approach to detection (set prediction with bipartite matching rather than region proposals), and understanding how plain ViT backbones interact with these end-to-end detectors would broaden the paper’s conclusions.
• The controlled pyramid comparison establishes overall AP improvements; it does not prove that a final-map pyramid reproduces the same representations as a hierarchical backbone for every object size.

Impact and Legacy

This paper, known as ViTDet, became the standard approach for using plain ViT backbones in detection and influenced the broader trend toward simpler, non-hierarchical architectures for dense prediction tasks. It demonstrated that the complexity of hierarchical vision transformers (Swin’s shifted windows, MViTv2’s pooled attention) may be unnecessary when strong pre-training is available.

The work directly informed Meta’s Segment Anything Model (SAM), which uses a ViTDet-style backbone for its image encoder. SAM’s success in promptable segmentation provided further evidence that plain ViT backbones are sufficient for dense visual tasks when properly pre-trained. The ViTDet approach also appears in DINOv2 and other foundation model pipelines where a single pre-trained backbone is adapted to multiple downstream tasks.

The paper also influenced how the community thinks about the relationship between pre-training and architecture. The finding that MAE pre-training narrows the gap between plain and hierarchical ViTs suggests a general principle: strong self-supervised pre-training can compensate for missing architectural inductive biases. This has practical implications for the model development lifecycle — rather than engineering task-specific architectures, teams can invest in pre-training a single plain ViT and adapt it to diverse downstream tasks (classification, detection, segmentation, depth estimation) through lightweight task-specific heads.

The ViTDet recipe (plain ViT backbone + windowed attention + simple feature pyramid) has been adopted beyond COCO object detection. It has been applied to video understanding, medical image analysis (where hierarchical features were previously considered essential for multi-scale pathology detection), and 3D point cloud processing. In each case, the core lesson holds: a plain transformer with minimal task-specific adaptation can match or exceed purpose-built architectures when pre-training is sufficiently strong.

The paper also contributed to the emerging consensus that foundation models benefit from architectural simplicity. A plain ViT backbone can be pre-trained once (with MAE, DINO, or similar self-supervised methods) and then deployed across classification, detection, segmentation, and generation tasks with only lightweight adapters. This amortizes the substantial pre-training cost across many downstream applications, a model that becomes increasingly attractive as pre-training datasets and compute budgets grow.

More broadly, the paper contributed to a reassessment of architectural inductive biases in computer vision. The results suggest that with sufficient pre-training data and appropriate self-supervised objectives, learned representations can substitute for architectural priors like multi-scale feature hierarchies — echoing a recurring theme in deep learning where scale and data diminish the value of hand-designed structure.

Related Reading

• Vision Transformer (ViT) — the plain ViT architecture that serves as the backbone in this work
• Attention Is All You Need — the original transformer, whose self-attention mechanism underpins ViT
• DINO — self-supervised ViT pre-training that provides complementary pre-training strategies for dense prediction
• Data Movement Is All You Need — analysis of the memory-bound operations that dominate transformer inference, relevant to ViTDet’s efficiency considerations