Deformable DETR: Learn Where to Read

The bottleneck is reading the image

DETR asks learned object queries to predict a set of detections. Its matching loss decides which slot should explain each object, but that does not tell a new query where to look. Dense attention must discover useful image locations while spreading its initial attention over many alternatives. Increasing feature resolution gives small objects more spatial detail while also making image attention more expensive.

Deformable DETR changes that access pattern. A query predicts a small collection of sampling offsets and aggregation weights around a reference coordinate. It reads those locations across several feature resolutions, mixes the sampled values, and updates its representation. The method keeps set prediction while making sparse, spatially structured reads the building block of image attention. Decoder self-attention between object queries remains dense.

The first preprint appeared in October 2020; the paper was presented at ICLR 2021. This review follows the March 2021 v4 paper. All interactive scenes below are authored deterministic illustrations, with calculated quantities where arithmetic is shown. The results panel separately labels published measurements. There are no recorded pretrained detections or newly measured runtimes on this page.

For one query and one head, dense image attention visits every spatial key. Deformable attention requests LK sample locations, where L is the number of feature levels and K is the number of points per level. That budget is independent of the total number of pixels. It does not eliminate the cost of projecting feature values, predicting offsets, combining heads, or interpolating fractional reads. The figure counts sampling locations, not individual memory loads or milliseconds.

This distinction also separates the encoder and decoder costs. Dense image self-attention has a quadratic spatial interaction count; query-to-image cross-attention has a query-count times spatial-count interaction count. Replacing their image reads with sparse sampling addresses both patterns, but does not make the whole detector constant-time. See §4.1 and Appendix A.1 for the module and complexity derivation.

A reference links resolutions without equating their pixels

A small object may occupy several cells on a fine map and less than a cell on a coarse map. Reading multiple resolutions gives a query access to different spatial support. Deformable DETR forms a feature pyramid from backbone features and additional projected/downsampled features; it does not require a top-down FPN fusion path to combine those levels. Fusion happens inside attention.

The bridge between levels is a normalized reference point \hat p_q. Its coordinates refer to the valid image, not an identical integer cell index in every tensor. Each level can have a different padded width and height. If its unpadded fractions are r_l=(W_l^{valid}/W_l,H_l^{valid}/H_l), then a point reference maps into its padded sampling frame as

Division is componentwise: horizontal offsets divide by width and vertical offsets by height. In the encoder, a source cell center is first divided by that source level's valid extent, then multiplied by each target level's valid ratio. In the decoder, valid-image references are multiplied by target-level ratios before calling attention. The diagram exposes that target mapping; it does not generate an encoder feature tensor.

Padding is more than a coordinate convention. The official attention module also zeroes projected values at masked positions. A read near a valid-image edge can therefore mix valid features with masked zeros even when its coordinates are inside the padded tensor. Values beyond the tensor are handled by the sampler's separate zero-padding rule. Both matter; valid-ratio scaling alone does not mask features. These details are visible in the official reference-point and valid-ratio construction.

The scalar helpers use shape order [width, height] for readable coordinate arithmetic. Upstream stores spatial shapes as [height, width] and explicitly reverses them when forming its offset normalizer. Confusing the two silently changes samples on non-square feature maps.

A learned location can fall between cells

Predicting a fractional coordinate is useful only if the read changes smoothly with that coordinate. Bilinear interpolation blends the surrounding cell values. It is piecewise differentiable with respect to the location, letting training adjust offsets rather than round them to integer indices. The browser scene calculates this arithmetic on a tiny authored grid; it is not an attention heatmap from a checkpoint.

The official PyTorch reference maps normalized sample locations s into grid_sample coordinates 2s-1, uses align_corners=False, and supplies zero padding. In pixel-index coordinates, this gives

Thus (u,v)=(0,0) lies half a pixel before the first pixel center along each axis. Pixel (i,j) has normalized center ((i+\tfrac12)/W,(j+\tfrac12)/H). The interpolated scalar is

where out-of-bounds values are zero and (a)_+=\max(a,0). Only four neighboring cells have nonzero weight. Do not clamp a sample to the nearest edge, and do not renormalize the remaining weights when some neighbors are outside.

The scene makes the boundary case visible: at the image corner, three neighbors are padding, so a first-cell value of 1 contributes only 1/4. Far enough outside, every contributing neighbor is zero. The actual CUDA operator and its PyTorch reference core process channel vectors and multiple heads; the scalar example isolates interpolation without implying a full operator reproduction.

Mix samples with a per-head budget

Sampling specifies where information can enter. Aggregation specifies how strongly each read contributes. For each query and head, learned linear projections of the query feature predict offsets and attention logits. The logits are normalized jointly across all levels and points within that head:

A different head gets its own distribution. With two levels and two equal-logit points at each level, each weight is 1/4. Applying an independent two-point softmax at each level would assign total mass 2, changing the operation.

The full multi-scale operator combines projected values, then combines heads:

Here W'_m selects a head's value subspace and W_m maps its aggregate into the output representation. The illustrated values stand in for already projected scalars. The attention coefficients are not computed by comparing the query with every sampled key: query features directly predict offsets and coefficients. Sampled content still enters through the values and affects later features. The official attention module makes these separate query projections explicit.

Sparse sampling also does not guarantee a local receptive field. Learned offsets can move away from the reference, and different heads can use different patterns. The reference is an organizing coordinate, not a hard crop. Nor are individual sample locations object detections: the decoder predicts class scores and a box from the resulting query representation.

Predict a box, then optionally feed it back

In the base one-stage detector, learned query embeddings supply content and positional parts. The positional part predicts a 2D reference point through a linear layer and sigmoid. Box prediction adds the reference's inverse-sigmoid coordinates to predicted center deltas, while predicting width and height without prior size logits:

This does not mean every object query starts from an anchor box. A point has no initial width or height. The optional iterative refinement variant lets a decoder layer's predicted box become the next layer's reference. Once the reference has four components, refinement adds all four previous box logits:

The released decoder detaches updated references before passing them onward. That boundary stops gradients through the reference update path; it does not remove every hidden-state gradient across decoder layers. The manual explorer draws a forward trace only, with no measured backward pass.

Box references also change sampling geometry. For a 4D reference already mapped into the target level's padded frame, the official attention module uses

Unlike the 2D branch, offsets are scaled by the reference box's size rather than divided by the feature-map dimensions. Before entering this formula, both center and size are scaled by the level's valid ratio. The same predicted offset can therefore mean a larger spatial displacement for a larger reference box. This box-relative path belongs to the relevant optional variants; it should not be retroactively assigned to every base query.

Two-stage initialization is a separate choice

Refinement asks how to update an existing reference. Two-stage initialization asks where the initial references come from. The encoder predicts candidate boxes and ranking scores at its spatial positions. The released proposal generator starts from grid centers normalized by valid dimensions and level-dependent size priors, predicts residual box coordinates, then gathers the highest-ranked proposals. It is proposal-based initialization, even though it does not use an NMS selection stage.

The selected box logits are detached, converted to initial 4D references, and positionally encoded. A learned transform of that encoding supplies the decoder's positional and content query parts. This differs from the one-stage learned query pair, and from later mixed-query designs that keep independently learned content while selecting spatial anchors. In the pinned implementation, the top-K ranking specifically reads encoder class channel zero; treating it as arbitrary maximum-over-class scoring would misdescribe this code path.

The proposal scene intentionally allows overlapping slots to survive ranking. Top-K does not perform overlap suppression. Set supervision and decoder interactions still have to discourage duplicate final predictions; the absence of NMS is not a proof that duplicates are impossible.

The results chart separates original v4 Table 1 values from later released-repository numbers. Both identify the base, refinement, and refinement-plus-two-stage configurations rather than attributing every improvement to sparse attention alone. Their training recipe uses 300 object queries and focal classification loss, unlike the original DETR setup. Feature levels, loss and schedule all matter when interpreting the comparison. See the paper experiment setup and official released results.

No latency comparison is plotted. The official timing notes use V100 GPUs and distinguish batch-1 inference from batch-4 throughput; precision and measurement boundaries must accompany any reused timing value. They cannot be joined to another detector's TensorRT/T4 result as if the workloads matched. Fewer training epochs likewise do not establish proportional wall-time savings.

From the equations to runnable arithmetic

The five rows below connect the paper and pinned implementation to a small Python reference. They cover coordinate mapping, interpolation, aggregation, reference updates and proposal ranking. The JavaScript diagrams use the same mathematical cases, with scoped tests checking Python parity. None of these examples train a detector or claim to reproduce published AP.

Each panel extracts its Python directly from the canonical offline example. The rows implement arithmetic subproblems, not a trained detector. The linked official implementation is pinned to revision 11169a6.

1. Translate a reference into each sampling frame

Original paper

Enlarge figure

Math and tensor shapes

s_{mlk}=\hat p\odot r_l+\Delta p_{mlk}/(W_l,H_l)

This is the 2D point branch. The Python helper also handles a 4D box: replace the offset term by (Δp/K) ⊙ (w,h) ⊙ r_l / 2. Its shape argument is [width, height], whereas the upstream spatial_shapes tensor stores [height, width]. Ratios assume rectangular right/bottom padding. The helper receives offsets; it does not learn them.

reference [2 or 4], offset [2], level size [2] → location [2]

Inspect corresponding official code

Python implementation

python

def sampling_location(reference, offset, shape, points, valid_ratio=(1, 1)):
    """Maps point or box references into a padded level's normalized frame.

    Args:
        reference: Valid-image coordinates [x, y] or [x, y, width, height].
        offset: Authored offset [dx, dy].
        shape: Padded level size [width, height].
        points: Number of samples per level per head.
        valid_ratio: Unpadded fraction [valid_width/W, valid_height/H].

    Returns:
        Two normalized sampling coordinates, with no clamping.
    """
    result = []
    for axis in range(2):
        center = reference[axis] * valid_ratio[axis]
        if len(reference) == 4:
            shift = (offset[axis] / points * reference[axis + 2]
                     * valid_ratio[axis] * 0.5)
        else:
            shift = offset[axis] / shape[axis]
        result.append(center + shift)
    return result

2. Resolve a non-integer location without clamping

Original paper

Enlarge figure

Math and tensor shapes

x=uW-\tfrac12,\quad y=vH-\tfrac12,\quad f(u,v)=\sum_{i,j} (1-|x-i|)_+(1-|y-j|)_+f_{ji}

Only the four floor/ceil neighbors can contribute. Out-of-bounds values are zero, and surviving weights are not renormalized. The normalized [0,1] sampling location becomes grid_sample coordinates 2s−1, with align_corners=false. Interior padded cells must already contain masked zeros; this helper does not accept a padding mask.

scalar grid [H, W], normalized (u,v) → scalar value

Inspect corresponding official code

Python implementation

python

def bilinear_sample(grid, u, v):
    """Samples a scalar grid with align_corners=False and zero padding."""
    height, width = len(grid), len(grid[0])
    x, y = u * width - 0.5, v * height - 0.5
    left, top = math.floor(x), math.floor(y)
    total = 0.0
    for row in (top, top + 1):
        for column in (left, left + 1):
            weight = (1 - abs(x - column)) * (1 - abs(y - row))
            if 0 <= row < height and 0 <= column < width:
                total += weight * grid[row][column]
    return total

3. Normalize levels and points within each head

Original paper

Enlarge figure

Math and tensor shapes

A_{mlk}=\frac{\exp a_{mlk}}{\sum_{l^\prime,k^\prime}\exp a_{ml^\prime k^\prime}},\quad h_m=\sum_{l,k}A_{mlk}v_{mlk}

The stable softmax subtracts one maximum across this head’s entire level/point array. Independent per-level softmax would incorrectly allocate one unit of mass to every level. These scalar values stand in for already projected value vectors; multi-head output projection is omitted. Real offsets and logits are separate linear projections of the query, not sampled-key compatibility scores.

one head: logits [L,K], values [L,K] → weights [L,K], scalar head output

Inspect corresponding official code

Python implementation

python

def normalize_weights(logits):
    """Normalizes one head jointly across levels and points."""
    maximum = max(value for level in logits for value in level)
    exp = [[math.exp(value - maximum) for value in level]
           for level in logits]
    total = sum(sum(level) for level in exp)
    return [[value / total for value in level] for level in exp]


def aggregate(values, weights):
    """Aggregates scalar samples after the value projection, for one head."""
    return sum(value * weights[level][point]
               for level, samples in enumerate(values)
               for point, value in enumerate(samples))

4. Keep the point and box logit branches distinct

Original paper

Enlarge figure

Math and tensor shapes

b^{\rm next}=\sigma(\Delta b+[\operatorname{logit}(p_x),\operatorname{logit}(p_y),0,0])

For a 4D reference, add logit(reference) to all four deltas instead. The implementation clips references near 0 and 1 for finite logits. The official decoder detaches updated references before passing them onward. This stdlib function checks forward arithmetic only; no gradient experiment or autograd is implied. Iterative reference feedback is optional, not the base detector.

reference [2 or 4], delta [4] → box [cx,cy,w,h]

Inspect corresponding official code

Python implementation

python

def refine_box(reference, delta):
    """Adds reference logits to centers only (2D) or all box axes (4D).

    This arithmetic does not implement the detached autograd boundary.
    """
    result = []
    for axis, change in enumerate(delta):
        if axis < len(reference):
            value = min(1 - 1e-5, max(1e-5, reference[axis]))
            change += math.log(value / (1 - value))
        result.append(1 / (1 + math.exp(-change)))
    return result

5. Rank encoder proposals before initializing the decoder

Original paper

Enlarge figure

Math and tensor shapes

I=\operatorname{TopK}(s),\quad r_0=\sigma(\operatorname{sg}(B_I))

B denotes unactivated box coordinates. The official code ranks encoder class channel zero, gathers box logits, detaches them and derives both query-position and content embeddings from the selected box positional encoding. This helper only ranks valid authored scores. It does not implement the encoder, proposal generation, positional transform, or detection evaluation; deterministic ID tie-breaking is an authored convention.

valid candidates [S], K → selected IDs [K]; no overlap suppression

Inspect corresponding official code

Python implementation

python

def top_proposals(proposals, count):
    """Ranks valid authored scores without NMS; ties use the slot ID.

    The official two-stage code ranks encoder class channel zero. This
    helper consumes scores already extracted from that ranking channel.
    """
    valid = [proposal for proposal in proposals if proposal['valid']]
    return sorted(valid, key=lambda item: (-item['score'], item['id']))[:count]

Download the Python, README and standard-library requirements file into one directory. The command prints deterministic JSON; no checkpoint, GPU or model download is needed.

Python sourceREADMERequirements

python3 deformable_detr.py --smoke

What this changes, and what comes next

Deformable DETR gives image attention an explicit spatial structure: choose a reference, request a small number of reads across levels, interpolate them, and combine them within a head. Optional box feedback and encoder proposals improve how those spatial coordinates evolve. This makes high-resolution information more practical without replacing set prediction with a hand-written duplicate-suppression pipeline.

The next difficulty concerns how object queries learn. Sparse reads do not themselves provide denoising supervision, guarantee stable assignment, or distinguish a useful spatial anchor from a useful content initialization. DINO detector addresses query training using dynamic-anchor and denoising ideas developed through intermediate work including DAB-DETR and DN-DETR. The reading sequence highlights that change of question; it is not a claim that one paper directly contains every later contribution.

For the original formulation, read Deformable DETR v4. For operational conventions, inspect the official repository alongside its pinned attention and transformer sources linked above. For set matching and the original query formulation, return to DETR.