An object query needs visual evidence before it can predict an object. With dense cross-attention, each query compares against every image-feature location. A finer map preserves more spatial detail, but also gives every query more locations to consider. Deformable attention replaces that dense image read with a small set of predicted sample locations.
The useful unit is one query reading one feature map: start from a reference point, predict offsets and weights, interpolate feature values at the resulting positions, then add the weighted samples. This lesson builds that operation before introducing multiple resolutions and heads. The reference point answers where the reads are organized; the query representation determines how those reads change.
Query content and spatial reference
An object query is a feature vector, not a pair of image coordinates. Its content evolves as decoder layers gather evidence. A reference point is a separate two-number spatial origin, usually represented in normalized coordinates p=(u_0,v_0). Horizontal coordinates run from left to right; vertical coordinates run from top to bottom.
Helpful background
Object queries
Distinguish learned prediction slots from image tokens and attention Q, then follow how decoder states produce detections.
A reference such as (0.5,0.5) identifies the map's middle. It does not specify an object's size, guarantee an object is present there, or require every sample to stay nearby. In the base one-stage Deformable DETR decoder, a learned positional query embedding is mapped through a linear layer and sigmoid to produce an initial point reference. The content vector and the point remain different quantities.
This lesson assumes an unpadded feature map. In a padded batch, the full detector also maps valid-image references into each level's padded coordinate frame and masks padded feature values. That bookkeeping is covered in the Deformable DETR paper walkthrough.
The query predicts where and how strongly to read
For one head and one map, projections of query features z_q predict K two-dimensional offsets and K attention logits. These projections are trained with the detector. The offsets can be fractional, large or out of bounds; the reference is an origin rather than a hard crop.
In the official point-reference implementation, offsets are expressed in feature-cell units. For a map of width W and height H, sample k is
A horizontal offset of one moves one feature-cell spacing. On a 4-column, 2-row map, offset (1,1) adds (1/4,1/2) in normalized coordinates. Width and height cannot be interchanged. Upstream stores shapes as [H, W] and explicitly constructs the offset normalizer as [W, H]. See the official point-reference branch.
Logits become positive aggregation weights through softmax. For one level,
Unlike dense query-key attention, this module directly predicts offsets and coefficients from the query features. It does not compute a query-key similarity for every image position. The sampled image content enters through feature values.
Fractional samples blend projected features
A sample often falls between feature cells. Rounding would turn a small offset change into either no change or a sudden jump. Bilinear interpolation instead blends the surrounding values, allowing the read to vary continuously with the location. The same basic interpolation idea appears in image resizing, but the sampled data here are projected feature vectors rather than RGB pixels.
The official reference sampler converts (u,v) to grid_sample coordinates (2u-1,2v-1) and uses align_corners=False with zero padding. In cell-index coordinates this means
Cell (i,j) therefore has normalized center ((i+\tfrac12)/W,(j+\tfrac12)/H). On the 4-by-2 map below, the first center is (0.125,0.25). The map boundary u=0 is half a cell before that first horizontal center.
For each of the four surrounding integer cells, multiply its value by its horizontal and vertical interpolation factors, then sum:
Out-of-bounds cells contribute zero. Their interpolation weight is retained: there is no edge clamping and no renormalization over the remaining cells. At (u,v)=(0,0), a first-cell value of 2 contributes only 2\times\tfrac12\times\tfrac12=0.5. A sample sufficiently far outside returns zero.
The explorer uses one scalar channel of an already projected 4-by-2 feature map. Change query content while leaving the reference fixed to see both predicted offsets and logits change. Then use the diagnostic controls to set S1's offsets to zero and move the reference to (0,0): the four terms show the corner's zero-padding calculation.
Diagnostic controls replace the selected sample’s predicted values. Choosing any query preset restores both query projections.
Query projection active: q = 0. Selected sample: S1.
- Head scalar sum
- 9.000
Amber cross: reference. Teal circle: S1. Teal square: S2. The selected marker is larger. Cells run left to right, then top to bottom.
| j / i | 0 | 1 | 2 | 3 |
|---|---|---|---|---|
| 0 | 2 | 4 | 6 | 8 |
| 1 | 10 | 12 | 14 | 16 |
S1 interpolation terms
- Cell (1, 0)0.500 × 4 = 2.000
- Cell (2, 0)0.000 × 6 = 0.000
- Cell (1, 1)0.500 × 12 = 6.000
- Cell (2, 1)0.000 × 14 = 0.000
Authored inputs and fixed affine coefficients; calculated interpolation and softmax. One head, one level, one already projected scalar channel. No trained model outputs or runtime measurements.
The query presets use authored affine coefficients, not trained parameters. They isolate the dependency from query content to sampling. Manual overrides replace the selected prediction so a particular coordinate or logit can be inspected. The displayed sums are calculated from those chosen values; they are not a recorded detector output.
Sampling and attention have different weights
Bilinear weights mix neighboring cells to obtain one sampled value. Attention weights mix those sampled values to obtain one head's result. Keeping these two operations separate prevents a common confusion: a sample with attention weight 0.8 can itself contain four bilinear contributions.
With query preset 0 and the default central reference, S1 reads (x,y)=(1,0.5), halfway between values 4 and 12, giving 8. S2 reads (2,0.5), halfway between 6 and 14, giving 10. Equal logits give equal attention weights, so the head's scalar result is
Changing a logit changes the balance between the samples without moving them. Moving a sample changes the interpolated value without changing its attention weight. If a sample moves fully outside the map, its value becomes zero while its attention weight remains in the softmax budget. The model must learn useful coordinates and useful coefficients together.
A real head samples a vector with several channels. The same bilinear coefficients apply to each channel, and each sample's attention coefficient scales its whole vector. The scalar result above represents one component before the final head-combining output projection.
Multiple resolutions and heads
A normalized reference supplies a common spatial origin across feature resolutions. Each level uses its own width and height to convert offsets into normalized displacement. Consequently, a one-cell offset on a coarse map spans a larger part of the image than a one-cell offset on a fine map. Feature pyramids explain why these resolutions provide different spatial support; multi-scale deformable attention mixes their sampled evidence directly.
For L levels and K samples per level, a head has LK logits. Its softmax runs jointly across levels and samples, not separately within each level. With two levels, two samples per level and equal logits, all four weights are 1/4. Two independent per-level softmax operations would incorrectly give total weight 2.
Each head predicts its own offsets and distribution, gathers its own value subspace, and produces an aggregate. The output projection combines the heads. The paper's multi-scale attention definition, §4.1, expresses that operation as
Here V_{m,l} denotes level l's projected values for head m, and W_m combines that head into the output. This is still a small read budget per query and head: LK locations rather than every spatial position. It does not remove feature projection, interpolation, prediction or memory-access costs. A sample count alone establishes no measured speedup.
From point references to the full detector
The name also recalls deformable convolution, which shifts a convolution kernel’s sampling sites at each output-grid position and combines those samples with kernel weights. Deformable attention instead organizes reads around each query’s reference and predicts query-conditioned aggregation coefficients. Both use learned sampling offsets; their output organization and weighting rules differ.
The operation now has a concrete chain: a query predicts offsets and coefficients, a point reference anchors the coordinates, bilinear interpolation supplies values, and weighted aggregation updates the representation. A sample location is a read location, not an object prediction. Class and box heads act on the updated query features.
A point reference has no width or height. Optional Deformable DETR variants can pass a four-component box reference, which scales offsets relative to box size instead of dividing them by the feature-map dimensions. That is a different coordinate branch; the point-reference formula in this lesson must not be applied unchanged to it.
Continue to Deformable DETR: Learn Where to Read for valid-region scaling, encoder and decoder usage, box references, iterative refinement, and proposal-based initialization. Those mechanisms build a detector around the sampling operation established here.
Related concepts
Understanding end-to-end object detection with transformers, from DETR's object queries to bipartite matching and attention-based localization
Follow spatial image features into DETR decoder slots, separating learned query embeddings, attention queries, changing content and object predictions.
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
