Skip to main content

How Object Detection Works

Understand object detection from image features to boxes, classes and scores. Compare one-stage and two-stage pipelines, training and duplicate handling.

Classification can say that an image contains a car. Detection must identify each car and locate it. Its output is therefore a variable-length list: each entry has a class, a bounding box, and a score. An image with no relevant objects should be allowed to produce an empty list.

Follow the prediction pipeline

Step through the two detector families below. Both reuse image features, but they organize object prediction differently. In the final step, notice that a wrong box can survive filtering: a high score and low overlap with other predictions do not prove that an object exists.

Two common detector pipelines
Detector family
Pipeline step

Extract features

A shared backbone converts image pixels into spatial feature maps. The scene below marks the input, not a recorded activation map.

Authored pipeline illustration; only the final thresholding and NMS are computed. One-stage versus two-stage describes the prediction pipeline, not whether boxes use anchors. DETR takes a different set-prediction approach.

Features, heads, and candidate boxes

A backbone produces spatial features from the image. A prediction head uses those features to estimate class evidence and geometry. Many detectors also combine multiple feature resolutions in a neck, such as a feature pyramid, to represent objects at different scales.

One-stage detectors make object predictions directly from image features without a separate proposal-conditioned classification stage. The original YOLO framed detection as predicting spatially organized boxes and class probabilities in one network evaluation. Modern descendants differ substantially in their details. YOLO paper

In a two-stage detector such as Faster R-CNN, a region proposal network suggests candidate regions. A second stage uses region features to classify and refine those proposals. The stages can share a backbone; “two-stage” does not mean processing the entire image with two independent CNNs. Faster R-CNN paper

One-stage versus two-stage is separate from anchor-based versus anchor-free prediction. The former describes pipeline organization; the latter concerns how candidate locations and box geometry are represented.

Training needs an assignment rule

An image may have three labeled objects and thousands of potential predictions. Training needs a rule for assigning predictions to targets, selecting positive examples, and representing background. The assignment may use anchor overlap, object centers, or a matching procedure, depending on the detector.

A simplified objective has classification and localization terms:

L=Lclass+λLboxL = L_{\text{class}} + \lambda L_{\text{box}}

This expression describes roles, not a universal implementation. A particular detector may also have objectness, quality, or other terms. Box loss is commonly applied to assigned positive predictions; the exact normalization and background handling matter when comparing implementations.

Inference needs a usable output list

A confidence threshold removes low-scoring predictions. Non-maximum suppression then processes candidates in score order and removes strongly overlapping alternatives of the same class. It tries to reduce duplicates, but in a crowded scene it can also suppress a nearby real object. Soft-NMS changes scores instead of immediately deleting every overlapping candidate.

NMS does not access ground-truth boxes. Evaluation does, using class-aware matching and an overlap criterion. Keep these two operations separate even when both use IoU. The detection metrics page shows the difference with explicit matches.

Where to go next

Learn box overlap before tuning thresholds. Then examine NMS and Soft-NMS, or RoI Pooling and RoI Align for the region-feature stage. Finally, DETR introduces set prediction and one-to-one training assignment, providing a different answer to duplicate handling.

If you found this explanation helpful, consider sharing it with others.

Mastodon