LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection

DINO detector makes object queries easier to initialize and train. A practical detector has another constraint: its full inference path must fit a small latency budget. Sparse image reads help, but backbone computation, feature layout, projection and decoder depth still consume time. LW-DETR asks how to organize that whole path while retaining the set-prediction formulation.

The result is a plain ViT encoder with interleaved local and global attention, depth aggregation, a C2f projector and three decoder layers. Stronger supervision and pretraining help recover accuracy within that runtime budget. These choices are separable: a layout optimization can change latency without changing the mathematical attention graph, while additional training supervision can improve accuracy without adding inference groups. The review follows those distinctions rather than attributing every gain to a smaller transformer. The source is the 2024 preprint, §§3–4; no conference venue is confirmed here.

Start with the whole inference path

A 640×640 image is split into 16×16 patches, giving a 40×40 encoder grid. Transformer blocks process that grid, intermediate and final representations are aggregated, and a projector adapts the resulting features for the detector. The decoder then updates a fixed set of object queries and predicts classes and boxes. The opening diagram makes the three computational stages explicit.

The tiny model uses six encoder blocks with width 192, a 256-channel projector and 100 queries. Small uses ten blocks at the same encoder width and 300 queries. Medium increases encoder width to 384 while retaining the single-scale projector. Large and XL use wider representations and a two-scale projector; XL increases encoder width to 768. All five use three decoder layers. Those are distinct configurations, not labels for an otherwise identical network. See Table 1.

This decomposition explains why reducing one component’s operations is insufficient. Attention may be cheaper while layout conversions remain expensive, and a six-layer decoder can take a substantial share of a tiny detector’s runtime. The paper reports a decoder reduction from 1.4 to 0.7 ms when moving from six to three layers, in its tiny-model setting. That observation motivates a shallow decoder; it is not a hardware-independent promise of a twofold speedup for the entire detector.

Local attention saves pairs; global blocks reconnect the image

A global attention head over N tokens evaluates N^2 ordered query/key pairs. Partitioning the grid into non-overlapping windows of side w reduces that block to Nw^2 pairs: each of the N queries reads the w^2 keys in its window. At 640×640 with patch side 16 and window side 10, N=1600, giving 160,000 local pairs versus 2,560,000 global pairs per head. This accounting includes self-pairs and excludes projections, FFNs and other work.

Fixed windows alone isolate regions. LW-DETR follows ViTDet’s interleaving strategy: in the six-block tiny encoder, blocks 1, 3 and 5 use window attention, while blocks 2, 4 and 6 use global attention. Global blocks carry information across boundaries without requiring every block to be global. The figure shows structural reach through an authored miniature grid; it does not show measured attention probabilities. Section 3.4 describes the actual layout and alternation.

Once a global block has contextualized every token, a later local block can still depend on the original full image through those contextualized inputs. Its immediate keys remain local. Keeping immediate reads separate from accumulated receptive field prevents the mistaken conclusion that a later window block erases global information.

The operation reduction also creates a systems question: how are the window tokens laid out in memory? That question leads to the next change.

Window-major layout removes repeated rearrangement

A spatial grid and a storage sequence are different objects. In row-major storage, neighboring rows of one window can be separated by tokens from other windows. Local attention commonly needs a permutation to collect those tokens. LW-DETR instead keeps the feature map organized window by window so both local and global blocks can consume that organization. The global block’s connectivity remains global.

For the paper’s 4×4 toy grid with 2×2 windows, row-major IDs 0, 1, 4 and 5 become the first contiguous window. The sequence then continues with 2, 3, 6 and 7. Every ID occurs exactly once, and the inverse permutation reconstructs the original grid. This is a storage transformation, not pooling, pruning, spatial displacement or a new attention mask. Positional information must stay associated with each token’s spatial identity.

The authors attribute the latency improvement to avoiding costly transitions between layouts. Our diagram verifies the permutation and its inverse; it does not verify a CUDA kernel or reproduce the deployment timings. The official backbone is the implementation reference. The distinction matters because mathematical equivalence alone does not establish the cost of data movement on a particular runtime.

Aggregate depths before creating spatial scales

The encoder produces a sequence of feature maps at one spatial resolution. Combining intermediate and final maps makes information from different representation depths available to the detector. Calling those maps “multi-level” does not mean they already form a multi-resolution pyramid.

For tiny, small and medium, the C2f projector emits a single stride-16 map with 256 channels. Large and XL change the projector: a deconvolution upsamples the aggregated input to stride 8, and a strided convolution downsamples it to stride 32. Two parallel C2f blocks process those branches, each emitting 384 channels, and the decoder uses multi-scale deformable cross-attention. At 640×640, these are 80×80 and 20×20 maps. The 40×40 encoder grid itself has not become hierarchical. Figure 3 and §3.1 separate the two projector paths.

This separation also explains the variant boundary. A tiny or small detector should not be illustrated as consuming the large model’s two scales. Increasing encoder width, fusing depths and creating projector scales are different architectural choices with different consequences for feature content and cost.

Keep inference shallow; strengthen training separately

Each of the three decoder layers contains query self-attention, deformable cross-attention to projected image features and an FFN. The query initialization follows mixed query selection: content comes from learned embeddings, while spatial information comes from top-K encoder-derived box proposals. This is an inherited query design, related to detector DINO; it is not a claim that LW-DETR introduced mixed queries.

Group DETR supplies additional supervision during training. Thirteen query groups use shared decoder weights, with a separate one-to-one matching problem for each group. A ground-truth object can therefore supervise a prediction in each group. Inference retains only the primary group, with 100 queries for tiny and 300 for the other variants. Thirteen groups do not imply thirteen decoder layers or thirteen inference passes. Section 3.3 explains this training/inference split.

Training also changes what a positive classification score should mean. For a matched positive, the IoU-aware target mixes current classification confidence s and box IoU u:

A poor-quality box receives a smaller positive target even when it has the right class. High-confidence unmatched negatives receive larger penalties than easy negatives. Geometry still has its own losses:

The figure evaluates the forward arithmetic in Eqs. 1–2. The pinned official criterion additionally floors positive targets at 0.01, detaches them and normalizes the reduced loss. The offline helper deliberately follows the printed equations without those implementation details; changing sliders is not a training or gradient experiment. The source/math/Python rows below make that boundary explicit.

Architecture and objective are only part of the recipe. The paper uses CAEv2 masked-image pretraining on Objects365 based on pretrained models, followed by supervised detector pretraining on Objects365, then COCO fine-tuning. Detector pretraining runs for 30 epochs; COCO fine-tuning uses 180K iterations with AdamW and EMA decay 0.9997. Tiny/small/medium use batch 32, making that 50 epochs; large/XL use batch 16, making it 25 epochs. Consequently, the same iteration count is not the same number of epochs across variants. Pretraining is essential context for the reported accuracy, rather than a detail to hide behind an architecture diagram. See §§3.3 and 4.1.

Read the evidence with its deployment conditions

The most useful systems result is the ablation where operation count and latency move in opposite directions. After depth aggregation, the small-model baseline uses 23.0 GFLOPs at 3.7 ms. Interleaving local/global attention reduces operations to 16.6 GFLOPs but increases latency to 3.9 ms in the compared row-major implementation. Window-major organization retains 16.6 GFLOPs and 34.7 AP while bringing latency to 2.9 ms. These adjacent rows support a claim about this implementation and measurement protocol, not a universal mapping from FLOPs to milliseconds. Table 2 contains the full cumulative ablation.

The paper measures COCO val2017 at 640×640 on a T4 with FP16, batch 1, TensorRT 8.6.1, CUDA 11.6 and cuDNN 8.7.0. Its cross-family end-to-end comparisons include NMS for non-DETR methods and distinguish official NMS thresholds from tuned thresholds. Those boundaries affect both latency and AP. A PyTorch forward pass, another GPU or an authored pair count cannot reproduce that comparison. Nor should these points be connected to DINO’s A100 timings as one common frontier.

The final tiny/small paper results are 42.6/48.0 AP. The official released results distinguish reproduced AP of 42.9/48.1. Keep those result sets separate. Checkpoint filenames containing “60e” do not automatically establish equivalence to the paper’s 180K-iteration schedule. Table 2’s earlier ablations use 45K iterations, while its final row uses 180K; the full progression also includes loss changes, grouped supervision, box reparameterization and Objects365 pretraining. It cannot isolate “the architecture alone” as the cause of the final accuracy.

Follow the math into runnable Python

The reference uses only Python’s standard library and prints deterministic JSON. It implements permutation, structural reach, output geometry, query budgets and classification arithmetic. It does not implement a detector, load a checkpoint, train, estimate AP or measure GPU latency. Source excerpts are synchronized from the canonical example rather than copied into the page by hand.

The five regions below come from one canonical standard-library Python source. They are authored calculations, not trained detector outputs. Read the original source beside each formula; then run the downloadable example to inspect the same fixture offline.

1. A permutation changes addresses, not features

Original paper

Enlarge figure

Math and tensor shapes

\pi = [0,1,4,5,2,3,6,7,8,9,12,13,10,11,14,15]

4×4 spatial grid → 16 token IDs; four contiguous 2×2 windows

For a 4×4 grid, token 4 moves from address 4 to address 2. Restoring by token ID exactly reconstructs row-major values. Coordinates and positional information travel with the token. The official backbone handles its own tensor and position layouts; this helper only teaches address order.

Pinned official implementation · backbone/vit.py

Python implementation

python

def window_order(rows, cols, side):
    """Return row-major token IDs in window-major storage order."""
    if any(not isinstance(v, int) or isinstance(v, bool) or v <= 0
           for v in (rows, cols, side)):
        raise ValueError("Expected positive integer dimensions")
    if rows % side or cols % side:
        raise ValueError("Grid must be divisible by window side")
    return [(y + dy) * cols + x + dx
            for y in range(0, rows, side)
            for x in range(0, cols, side)
            for dy in range(side) for dx in range(side)]


def restore_order(values, order):
    """Undo a bijective storage permutation without changing token values."""
    if len(values) != len(order) or sorted(order) != list(range(len(order))):
        raise ValueError("Expected a permutation")
    restored = [None] * len(values)
    for index, token_id in enumerate(order):
        restored[token_id] = values[index]
    return restored

2. Count edges and propagate structural reach

Original paper

Enlarge figure

Math and tensor shapes

P_{\rm local}=Nw^2,\quad P_{\rm global}=N^2

N=16, w=2 → 64 local pairs or 256 global pairs per head

Here w is the window side, so each window has w² tokens. Counts include self-attention and ordered query/key pairs. Consecutive local blocks cannot cross a window boundary; one intervening global block gives every output structural access to the full original grid. These counts omit channels, FFNs, projections and data movement, and establish no latency.

Pinned official implementation · backbone/vit.py

Python implementation

python

def attention_pairs(rows, cols, side):
    """Count ordered query/key pairs per head, including self-attention."""
    window_order(rows, cols, side)
    count = rows * cols
    return {"local": count * side * side, "global": count * count}


def reachable_tokens(rows, cols, side, selected, layers):
    """Compute structural reach, not learned attention weights."""
    window_order(rows, cols, side)
    if not isinstance(selected, int) or not 0 <= selected < rows * cols:
        raise ValueError("Invalid token")
    reached = {selected}
    for layer in layers:
        if layer == "global":
            reached = set(range(rows * cols))
        elif layer == "local":
            reached = {
                ((token // cols) // side * side + dy) * cols
                + (token % cols) // side * side + dx
                for token in reached for dy in range(side) for dx in range(side)
            }
        else:
            raise ValueError("Unknown attention kind")
    return sorted(reached)

3. Separate encoder depths from projector scales

Original paper

Enlarge figure

Math and tensor shapes

H_{16}=H/16,\quad H_8=2H_{16},\quad H_{32}=H_{16}/2

640 input → 40² encoder positions; large/XL → 80² and 20² outputs

Tiny, small and medium emit one stride-16 map with 256 channels. Large and xlarge emit stride-8 and stride-32 maps with 384 channels each, via deconvolution and strided convolution followed by parallel C2f blocks. This helper describes exact divisible geometry; it does not execute depth fusion, C2f or learned upsampling.

Pinned official implementation · backbone

Python implementation

python

def projector_geometry(variant, resolution):
    """Describe output geometry; no C2f or feature fusion is executed."""
    if variant not in ("tiny", "small", "medium", "large", "xlarge"):
        raise ValueError("Unknown variant")
    large = variant in ("large", "xlarge")
    divisor = 32 if large else 16
    if not isinstance(resolution, int) or resolution <= 0 or resolution % divisor:
        raise ValueError("Resolution must divide exactly")
    return [{"stride": stride, "side": resolution // stride,
             "channels": 384 if large else 256}
            for stride in ((8, 32) if large else (16,))]

4. Budget query groups without duplicating inference

Original paper

Enlarge figure

Math and tensor shapes

Q_{\rm train}=13K,\quad Q_{\rm infer}=K,\quad L_{\rm decoder}=3

Tiny: 1,300 training slots → 100 inference slots; others: 3,900 → 300

Each group solves its own one-to-one matching problem, giving each target multiple training supervision opportunities across groups. The helper counts slots; it does not initialize proposals, compute matching or implement group attention isolation. Thirteen is a training multiplicity, not the decoder depth or thirteen inference passes.

Pinned official implementation · lwdetr.py

Python implementation

python

def query_budget(variant, training):
    """Count Group DETR query slots sharing decoder weights."""
    if variant not in ("tiny", "small", "medium", "large", "xlarge"):
        raise ValueError("Unknown variant")
    per_group = 100 if variant == "tiny" else 300
    groups = 13 if training else 1
    return {"groups": groups, "perGroup": per_group,
            "total": groups * per_group, "layers": 3}

5. Compute the paper’s IoU-aware forward objective

Original paper

Enlarge figure

Math and tensor shapes

t=s^{0.25}u^{0.75},\quad \ell=\sum_+\operatorname{BCE}(s,t)+\sum_-s^2\operatorname{BCE}(s,0)+2\ell_{\rm GIoU}+5\ell_1

s=0.16, u=0.81 → t≈0.54; negative s=0.5 → 0.25 log(2)

BCE(s,t)=−t log(s)−(1−t)log(1−s). The pinned official criterion floors t at 0.01, detaches it (including IoU from detached boxes), and reduces/normalizes losses. This reference does not reproduce those details, gradients, matching, distributed normalization or the box-loss terms. It computes one classification contribution with natural logarithms and explicit endpoint conventions.

Pinned official implementation · lwdetr.py

Python implementation

python

def iou_target(score, iou):
    """Evaluate the paper's positive target t=s^.25*u^.75."""
    if any(not math.isfinite(v) or not 0 <= v <= 1 for v in (score, iou)):
        raise ValueError("Expected probability in [0,1]")
    return score ** 0.25 * iou ** 0.75


def ia_bce(score, iou, positive):
    """Return paper-equation forward loss, without autograd or matching.

    Endpoints use extended-real BCE with 0*log(0)=0. The official criterion
    additionally controls detached targets, normalization and reduction.
    """
    target = iou_target(score, iou) if positive else 0.0
    if not math.isfinite(score) or not 0 <= score <= 1:
        raise ValueError("Expected probability in [0,1]")
    positive_term = (0.0 if target == 0 else
                     math.inf if score == 0 else -target * math.log(score))
    negative_term = (0.0 if target == 1 else
                     math.inf if score == 1 else
                     -(1 - target) * math.log1p(-score))
    return (1 if positive else score * score) * (positive_term + negative_term)

Download the Python reference, its README, and the requirements receipt. Run python3 lw_detr.py; no third-party installation is needed. These are authored calculations. This review contains no recorded pretrained inference or newly measured benchmark.

The next question: which configuration should we deploy?

LW-DETR shows how a plain ViT, careful storage organization, depth aggregation, a suitable projector and a shallow decoder can form a practical set predictor. Its training recipe makes that small inference graph more effective. The window permutation changes execution organization; Group DETR changes training supervision; neither should be confused with a new detector output recorded in this page.

The remaining choices are still fixed by a model configuration: backbone, feature geometry, query budget and capacity. RF-DETR takes LW-DETR as an architectural starting point, replaces the backbone with self-supervised DINOv2 features and explores adaptation to task and hardware. Detector DINO and self-supervised DINOv2 are distinct contributions. That next review asks which operating point to choose, building on the separation here between architecture, training and measured deployment behavior.

Primary references

• LW-DETR paper v1: architecture and objective (§3.1), variants (Table 1), training (§3.3), layout (§3.4), ablation (Table 2) and experimental conditions (§4.1).
• Official LW-DETR repository: released results, configuration/checkpoint links and deployment workflow. Source correspondence here is pinned to commit d5e6e6c4add2d24dafb965ced8b50163c50b9788.
• Group DETR and ViTDet: grouped training and the interleaved attention context inherited by this design.