Skip to main content

DETR Explained: Object Detection with Transformers

Understanding end-to-end object detection with transformers, from DETR's object queries to bipartite matching and attention-based localization

Overview

Modern object detection has evolved from complex multi-stage pipelines (R-CNN family) and anchor-based single-shot detectors (YOLO, SSD) to elegant transformer-based architectures. DETR (DEtection TRansformer) pioneered this shift by treating object detection as a direct set prediction problem, eliminating hand-designed components like anchor boxes, non-maximum suppression (NMS), and region proposal networks.

The key innovation is using learned object queries to distinguish decoder prediction slots that attend to image features. Combined with bipartite matching during training, DETR predicts a set of detections without an NMS duplicate-suppression stage. Scores and normalized box coordinates still need interpretation for evaluation or display.

DETR Architecture Walkthrough

Step through DETR's pipeline one stage at a time — from raw pixels to a set of predictions — watching the tensor shapes and operation at each stage. A single forward pass replaces the multi-stage anchor-and-NMS machinery of earlier detectors.

DETR architecture
Stage
Stage
1 / 6
Tensor shape
B×3×800×1066
This stage
raw pixels
Operation
Image input

Input Image

H × W × 3 (e.g., 800 × 1066 × 3)

Input Processing

  • RGB image of arbitrary size
  • Resized with max dimension 1333px
  • Normalized with ImageNet mean/std
  • Batched with padding if needed

# No region proposals needed! # No anchor boxes! # Just raw image pixels

Stage 1 of 6: Input Image

DETR's Key Innovation: By treating object detection as a set prediction problem with bipartite matching, DETR eliminates hand-designed components like anchor boxes, NMS, and region proposals — achieving a truly end-to-end detection pipeline.

Key Concepts

Object Queries

100 learned embeddings that specialize to detect different objects, positions, and scales during training

Set-Based Prediction

Predict all objects simultaneously without sequential decoding or duplicate suppression

Bipartite Matching

Hungarian algorithm optimally assigns predictions to ground truth during training

Cross-Attention

Object queries attend to encoder features to gather visual information for detection

End-to-End Training

No anchor boxes, NMS, or region proposals - just image in, detections out

Global Reasoning

Transformer self-attention enables reasoning about all objects and their relationships

Object Query Attention Visualization

Pick a query and scrub through an authored illustration of cross-attention sharpen — from a diffuse scan of the whole image into a focused lock on a single object.

Object query attention
Decoder layer
Object query

Decoder layer 1 of 6

Low attentionHigh attention
Query
#0
Predicts
person
Confidence
95%
Decoder layer
1 / 6
Hovered cell
—
Attention
—

Authored illustration: The attention weights, classes, confidence scores, and boxes are invented; no model or image was evaluated. The broad-to-sharp layer schedule is a chosen construction, not a measured result.
Illustrated Attention Evolution Through Layers
Early layers (1-2): The authored map is broad and diffuse.
Middle layers (3-4): The authored map becomes more focused.
Late layers (5-6): The authored map is sharper around its chosen focus.

Understanding Object Queries

Helpful background

Object queries

Distinguish learned prediction slots from image tokens and attention Q, then follow how decoder states produce detections.

Open full page

Object queries are learned slot embeddings. In the original implementation, decoder content states start at zero and the learned embeddings provide query-position information. Cross-attention lets each slot read image features, while self-attention lets the evolving slot states exchange information.

Each final slot state produces class scores and a box. Slots can develop spatial or scale preferences, but they have no fixed class identity. One-to-one matching and the set loss train distinct predictions; self-attention alone does not guarantee uniqueness. Unmatched slots receive the no-object classification target. Original decoder, prediction heads and training criterion.

Bipartite Matching for Training

Step through how the Hungarian algorithm pairs DETR's predictions with the ground-truth objects, turning detection into a one-to-one assignment problem that needs no duplicate suppression.

Bipartite matching in DETR
Step

Initial State

DETR outputs N=100 predictions (showing 8 for clarity)

Authored illustration: classes, probabilities, and normalized boxes are invented. All eight slots enter the cost matrix, including those currently predicting no object. Cost = −p(target class) + 5 × L1 (center, width, height) − 2 × GIoU; this matching cost is distinct from the training loss.

Step
1 / 5
Matched
—
Unmatched
—
Total cost
—

Predictions (N=100, showing 8)

  • Q0persontop class probability: 95%target probabilities: person 0.95, dog 0.01, car 0.01; box [0.12, 0.15, 0.38, 0.88]
  • Q1dogtop class probability: 87%target probabilities: person 0.02, dog 0.87, car 0.02; box [0.52, 0.48, 0.78, 0.92]
  • Q2cartop class probability: 91%target probabilities: person 0.02, dog 0.02, car 0.91; box [0.58, 0.08, 0.95, 0.42]
  • Q3persontop class probability: 23%target probabilities: person 0.23, dog 0.1, car 0.1; box [0.7, 0.6, 0.85, 0.95]
  • Q4cattop class probability: 15%target probabilities: person 0.1, dog 0.1, car 0.1; box [0.1, 0.7, 0.3, 0.9]
  • Q5birdtop class probability: 8%target probabilities: person 0.04, dog 0.04, car 0.04; box [0.4, 0.2, 0.5, 0.35]
  • Q6no objecttop class probability: 92%target probabilities: person 0.02, dog 0.02, car 0.02; box [0.05, 0.05, 0.2, 0.2]
  • Q7no objecttop class probability: 88%target probabilities: person 0.03, dog 0.03, car 0.03; box [0.8, 0.75, 0.95, 0.95]

Ground Truth (M=3)

  • G0personbbox: [0.14, 0.18, 0.36, 0.85]
  • G1dogbbox: [0.54, 0.50, 0.76, 0.90]
  • G2carbbox: [0.60, 0.10, 0.93, 0.44]

Key Insight: This exhaustive solver finds the global minimum for eight authored slots; DETR uses linear assignment. Each ground truth is assigned to exactly one prediction, enabling set-based training without NMS or anchor boxes.

Bipartite Matching Loss

Helpful background

Set prediction and matching

Find a one-to-one assignment with minimum total cost and distinguish training assignment from detection evaluation and NMS.

Open full page

Helpful background

NMS and Soft-NMS

Use IoU to follow duplicate suppression and compare removing boxes with decaying their scores.

Open full page

Traditional detectors assign multiple predictions to each ground truth (via IoU thresholds), then suppress duplicates with NMS. DETR takes a fundamentally different approach:

  1. Cost Matrix: Compute pairwise costs between all predictions and ground truth objects
  2. Hungarian Algorithm: Find optimal 1-to-1 assignment minimizing total cost
  3. Loss Computation: Matched pairs contribute class and box losses; unmatched slots contribute no-object classification loss

The cost combines classification probability and bounding box distance (GIoU + L1):

Helpful background

Box regression and GIoU

Compare coordinate losses with overlap-based objectives and inspect the enclosure term when boxes do not overlap.

Open full page
text
L_match(y_i, ŷ_σ(i)) = -1_{c_i≠∅} p̂_σ(i)(c_i) + 1_{c_i≠∅} L_box(b_i, b̂_σ(i))

Real-World Applications

Autonomous Driving

End-to-end detection of vehicles, pedestrians, and obstacles without complex post-processing

Tesla's vision-only approach uses transformer-based detection

Medical Imaging

Detect tumors, lesions, and anatomical structures in X-rays, CT, and MRI scans

Global context helps reason about rare pathologies

Satellite & Aerial Imagery

Detect buildings, vehicles, and infrastructure across large-scale images

Handles varying object densities without anchor tuning

Robotics & Manipulation

Real-time object detection for grasping and navigation

Deformable DETR achieves real-time performance

Video Understanding

Track objects across frames using query propagation

TrackFormer extends DETR for multi-object tracking

Document Analysis

Detect text regions, tables, and figures in documents

Layout analysis benefits from global context

This teaching sketch uses learned embeddings directly as decoder content and adds image positions once before the transformer. Original DETR instead starts decoder content at zero and injects image and query positional embeddings into attention queries and keys throughout its layers. The sketch illustrates the feature-to-prediction pipeline; it does not reproduce the original decoder. See the official transformer implementation for those details.

Simplified DETR Implementation

python
import torch
import torch.nn as nn
from torchvision.models import resnet50

class SimpleDETR(nn.Module):
    def __init__(self, num_classes=91, num_queries=100, hidden_dim=256):
        super().__init__()

        # CNN Backbone (ResNet-50, remove final layers)
        backbone = resnet50(pretrained=True)
        self.backbone = nn.Sequential(*list(backbone.children())[:-2])
        self.conv = nn.Conv2d(2048, hidden_dim, 1)  # Reduce channels

        # Transformer
        self.transformer = nn.Transformer(
            d_model=hidden_dim,
            nhead=8,
            num_encoder_layers=6,
            num_decoder_layers=6,
            dim_feedforward=2048,
            batch_first=True
        )

        # Object queries (learned embeddings)
        self.query_embed = nn.Embedding(num_queries, hidden_dim)

        # Prediction heads
        self.class_head = nn.Linear(hidden_dim, num_classes + 1)  # +1 for "no object"
        self.bbox_head = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 4),
            nn.Sigmoid()  # Normalized coordinates
        )

        # Positional encoding
        self.pos_encoder = PositionalEncoding2D(hidden_dim)

    def forward(self, x):
        # Extract features: (B, 3, H, W) -> (B, 2048, H/32, W/32)
        features = self.backbone(x)
        features = self.conv(features)  # (B, 256, H/32, W/32)

        B, C, H, W = features.shape

        # Add positional encoding
        pos = self.pos_encoder(H, W).to(x.device)
        features = features + pos

        # Flatten spatial dimensions: (B, 256, H*W) -> (B, H*W, 256)
        src = features.flatten(2).permute(0, 2, 1)

        # Object queries: (num_queries, 256) -> (B, num_queries, 256)
        queries = self.query_embed.weight.unsqueeze(0).repeat(B, 1, 1)

        # Transformer: encode image, decode with queries
        output = self.transformer(src, queries)  # (B, num_queries, 256)

        # Predictions
        class_logits = self.class_head(output)  # (B, 100, num_classes+1)
        bbox_pred = self.bbox_head(output)       # (B, 100, 4)

        return {'pred_logits': class_logits, 'pred_boxes': bbox_pred}

class PositionalEncoding2D(nn.Module):
    def __init__(self, hidden_dim, temperature=10000):
        super().__init__()
        if hidden_dim <= 0 or hidden_dim % 4:
            raise ValueError("hidden_dim must be a positive multiple of 4")
        self.hidden_dim = hidden_dim
        self.temperature = temperature

    def forward(self, h, w):
        y_embed = torch.arange(h).unsqueeze(1).repeat(1, w)
        x_embed = torch.arange(w).unsqueeze(0).repeat(h, 1)

        dim_t = torch.arange(self.hidden_dim // 4)
        dim_t = self.temperature ** (2 * dim_t / (self.hidden_dim // 2))

        pos_x = x_embed.unsqueeze(-1) / dim_t
        pos_y = y_embed.unsqueeze(-1) / dim_t

        pos_x = torch.stack([pos_x.sin(), pos_x.cos()], dim=-1).flatten(-2)
        pos_y = torch.stack([pos_y.sin(), pos_y.cos()], dim=-1).flatten(-2)

        pos = torch.cat([pos_y, pos_x], dim=-1)
        return pos.permute(2, 0, 1).unsqueeze(0)  # (1, hidden_dim, H, W)

# Usage

model = SimpleDETR(num_classes=91)
image = torch.randn(2, 3, 800, 1066)
output = model(image)
print(f"Class predictions: {output['pred_logits'].shape}") # (2, 100, 92)
print(f"Box predictions: {output['pred_boxes'].shape}") # (2, 100, 4)

Advantages & Limitations

Advantages

  • ✓End-to-end training without hand-crafted components (anchors, NMS)
  • ✓Global reasoning via self-attention handles object relationships
  • ✓Excellent performance on large objects due to global context
  • ✓Simpler pipeline with fewer hyperparameters to tune
  • ✓Easily extended to panoptic segmentation and tracking
  • ✓Set-based prediction naturally handles varying object counts

Limitations

  • ×Slow training convergence (500 epochs vs 36 for Faster R-CNN)
  • ×Struggles with small objects due to coarse feature resolution
  • ×High memory usage from full attention over image features
  • ×Fixed number of queries limits maximum detections per image
  • ×Requires careful learning rate scheduling and data augmentation
  • ×Inference slower than optimized CNN-based detectors

Evolution: Deformable DETR and Beyond

The original DETR's limitations sparked rapid innovation:

The following published COCO val2017 small-object AP results use different training schedules and configurations, so they are not controlled comparisons.

Model and configurationEpochsSmall-object APSource
DETR, ResNet-5050020.5Deformable DETR, Table 1
Deformable DETR, ResNet-50, multi-scale5026.4Deformable DETR, Table 1
DINO, ResNet-50, 4 scales1232.0DINO, Table 1
RT-DETR, ResNet-507234.8RT-DETR, Table 2

Conditional DETR introduces conditional cross-attention, while DAB-DETR uses anchor-based queries.

Deformable DETR is particularly impactful: it replaces full attention with deformable attention that only attends to a small set of key sampling points, reducing complexity from O(N²) to O(N).

Best Practices

  • Use Deformable Attention: For practical applications, use Deformable DETR or its variants for faster training and better small object detection
  • Multi-Scale Features: Use FPN-style multi-scale features to capture objects at different sizes
  • Auxiliary Losses: Add prediction heads after each decoder layer for faster convergence
  • Strong Augmentation: Use aggressive data augmentation (random crop, scale jitter, color jitter) to compensate for slow convergence
  • Careful Learning Rate: Use lower learning rate for backbone (1e-5) and higher for transformer (1e-4)
  • Query Design: Consider anchor-based queries (DAB-DETR) or denoising training (DINO) for faster convergence

Mathematical Foundation

The detection loss combines classification and localization:

text
L(y, ŷ) = Σ_i [ λ_cls · L_cls(c_i, ĉ_σ(i)) + 1_{c_i≠∅} · (λ_L1 · ||b_i - b̂_σ(i)||_1 + λ_giou · L_giou(b_i, b̂_σ(i))) ]

Where:

  • σ is the optimal assignment from Hungarian matching
  • L_cls is classification cross-entropy, with a lower weight for the no-object class
  • L_giou is generalized IoU loss for better box regression
  • The loss weights are configuration choices; the original released loss uses unit classification weight, λ_L1=5 and λ_giou=2. The no-object classification term has its own relative weight.

The Hungarian matching cost:

text
C_match(y_i, ŷ_j) = -p̂_j(c_i) + λ_L1 · ||b_i - b̂_j||_1 + λ_giou · L_giou(b_i, b̂_j)

Further Reading

For the sampling mechanism behind Deformable DETR, start with deformable attention and reference points, then continue to the paper walkthrough.

If you found this explanation helpful, consider sharing it with others.

Mastodon