An image encoder turns pixels into useful features. Keeping those features at different spatial positions lets a detector ask where objects are. But the image grid has a fixed structure while the answer does not: one photograph may contain two objects and another none. How can a network turn the same kind of feature grid into a variable-size set of detections?
DETR reserves a fixed number of prediction slots. Each slot gathers evidence from the image and eventually proposes a class and a box. A slot can also say no-object, so having room for many predictions does not require the image to contain that many objects.
From spatial memory to prediction slots
Original DETR uses a convolutional backbone, projects its feature map into the transformer's channel width, and flattens the spatial grid into a sequence. Positional encodings preserve information about where each feature came from. A transformer encoder contextualizes these features; its output is the decoder's image memory. This is a sequence of spatial features, rather than one pooled image summary.
The decoder has a separate sequence of slots. Its length is chosen when constructing the model, independently of the number of image-memory positions. Each slot can read the whole valid image memory through cross-attention. A slot is therefore neither an image patch nor one particular cell of the feature map.
The original DETR experiments use 100 slots. This is a capacity limit, not a prediction that every image contains 100 objects. For one image and object categories, the output shapes are class logits and box coordinates. The extra class is no-object. These are architectural dimensions, not measurements of detection accuracy. DETR, Section 3.2 and implementation.
What does “query” mean here?
Three related things need separate names:
- A learned object-query embedding identifies a slot. It is a trainable vector reused across images at inference time.
- A decoder content state holds what that slot has inferred about the current image. It changes as the decoder processes the image.
- An attention query vector is computed inside an attention operation. It is compared with key vectors to determine which values to combine.
In original DETR's official implementation, the initial decoder content is all zeros. The learned object-query embeddings are passed separately as query_pos. They enter attention queries and keys as positional additions; they are not image patches copied into the decoder. At the first layer, distinct learned embeddings allow otherwise identical zero-content slots to ask different questions. The embeddings are learned across training images, while the content becomes image-dependent during each forward pass. Official transformer initialization and decoder.
For one cross-attention head, the distinction can be written schematically as:
Here is the current decoder content, contains learned slot embeddings, is image memory, and contains spatial positional encodings. is the actual projected attention query; is one ingredient used to form it. The decoder also contains residual updates, normalization and feed-forward layers, omitted from this equation. Attention itself is described in scaled dot-product attention.
These are the mechanics of original DETR. Later detector variants may introduce reference points, anchors or image-dependent query initialization. Those designs should not be read back into the original zero-content initialization.
Inspect identity, content and output
Select a slot, move from Initialize to Read memory to Predict, then change the scene. The slot's learned identity remains the same while its content and eventual prediction depend on the image. All four slots remain present even for the empty scene.
Spatial image memory (shared by every slot)
Each slot starts with zero content and its own learned query-position embedding.
Authored illustration: four slots and their outcomes are chosen for explanation. No attention weights, trained outputs or fixed slot specializations are implied.
The explorer is a semantic illustration, not a trained decoder. In particular, slot 1 is assigned the dog only for this example. A real learned slot is not permanently assigned a class such as dog, and its embedding is not a fixed bounding-box anchor. Learned slots may develop statistical preferences for locations or sizes without becoming hard-coded assignments.
Slots interact before sharing the same heads
Decoder self-attention lets the slots exchange information with one another. Decoder cross-attention lets them retrieve information from the image memory. This difference is about the source of keys and values: other slots in the first case, spatial image features in the second. Unlike an autoregressive text decoder, original DETR predicts its slots in parallel without a causal ordering between them.
After decoding, a shared linear classification head reads every slot state, and a shared box multilayer perceptron predicts its box. There is not a separate dog head for one slot and bicycle head for another. Applying the same heads to different states can produce different answers. Original DETR uses a sigmoid for normalized box coordinates in center-x, center-y, width, height form. Official prediction heads and forward pass.
The heads produce box coordinates even for a slot classified as no-object. Those coordinates do not constitute an object detection merely because they exist. During training, unmatched slots receive the no-object classification target; box regression supervises matched object slots. No-object describes an unused prediction slot, not a segmentation label assigned to every background pixel.
Why not predict the same object twice?
Allowing slots to communicate makes coordination possible. It does not mathematically prevent two slots from predicting the same object. Cross-attention can overlap, and attention weights are not an exclusive partition of the image.
The training objective supplies the missing incentive. DETR matches each labeled object to one prediction, supervises that prediction's class and box, and trains unmatched predictions toward no-object. A second prediction of the same labeled object cannot claim a second match to that same target. This one-to-one assignment and set loss discourage duplicates; they do not guarantee that a trained model never makes one. DETR, Section 3.1.
That raises the next question: which slot should learn which object when targets have no natural order? If an image contains a dog and a bicycle, the annotations do not say that the dog belongs to slot 1. Set prediction and bipartite matching follows this handoff from decoder outputs to a training assignment. For the complete architecture around these pieces, continue to DETR object detection.
Related concepts
Follow a query from a spatial reference through learned offsets, bilinear feature sampling and weighted aggregation, with exact non-square coordinate arithmetic.
Understanding end-to-end object detection with transformers, from DETR's object queries to bipartite matching and attention-based localization
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
