You Only Look Once: Unified, Real-Time Object Detection

TL;DR

YOLO recasts object detection as a single regression problem. Instead of the multi-stage propose-then-classify pipeline used by R-CNN and its variants, a single convolutional network predicts bounding box coordinates and class probabilities directly from the full image in one forward pass. The base model runs at 45 FPS with 63.4 mAP on VOC 2007 — 90x faster than the 0.5 FPS Fast R-CNN entry, with 6.6 points lower mAP. The speed comes from eliminating region proposals entirely: the network reasons globally over the image, trading some localization precision for a dramatic reduction in inference cost.

The Core Idea: Detection as Regression

Prior to YOLO, dominant detectors like R-CNN, Fast R-CNN, and Faster R-CNN operated in stages: generate region proposals, extract features from each, then classify and refine. Each stage introduced latency and complexity. Deformable Parts Models (DPM) used sliding windows with hand-crafted features, which was even slower.

YOLO takes a fundamentally different approach. The input image is divided into an S \times S grid (with S = 7 in the paper). Each grid cell predicts B bounding boxes (with B = 2) and C class probabilities (with C = 20 for VOC). Each bounding box prediction consists of 5 values: center coordinates (x, y) relative to the grid cell, width and height (w, h) relative to the full image, and a confidence score reflecting both objectness and localization quality.

The confidence score is defined as:

This means the network output is a single tensor of shape S \times S \times (B \times 5 + C), which for the VOC configuration is 7 \times 7 \times 30. The entire detection pipeline — feature extraction, bounding box prediction, and classification — collapses into one forward pass through a single network.

Architecture: 24 Conv Layers + 2 Fully Connected

The YOLO network is inspired by GoogLeNet but replaces Inception modules with simple reduction layers. The architecture consists of 24 convolutional layers for feature extraction followed by 2 fully connected layers for prediction. The first 20 convolutional layers are pretrained on ImageNet at half resolution (224 x 224), then the full network is fine-tuned on detection at 448 x 448.

The convolutional layers use alternating 1x1 reduction layers and 3x3 convolutional layers. The final output of the FC layers is reshaped into the 7 \times 7 \times 30 prediction tensor. Leaky ReLU activation (\alpha = 0.1) is used throughout except for the final layer, which uses linear activation.

The paper also introduces Fast YOLO, a smaller variant with only 9 convolutional layers and fewer filters per layer. Fast YOLO achieves 155 FPS while still reaching 52.7 mAP on VOC 2007 — demonstrating that the single-regression framework scales down gracefully.

Grid-Based Prediction

The grid design encodes a strong spatial prior: each grid cell is responsible for detecting objects whose center falls within it. This means the cell at row i, column j predicts boxes only for objects centered in that cell. At test time, the class-specific confidence for each box is:

This yields per-box, per-class scores. After thresholding, non-maximum suppression (NMS) removes duplicate detections. With S = 7 and B = 2, YOLO produces 7 \times 7 \times 2 = 98 bounding box predictions per image — orders of magnitude fewer than the ~2000 proposals from Selective Search used by R-CNN.

The Multi-Part Loss Function

YOLO is trained end-to-end with a single sum-of-squared-errors loss that combines localization, confidence, and classification terms:

Three design choices in this loss are worth noting:

1. Square-root width/height: The loss uses \sqrt{w} and \sqrt{h} rather than raw dimensions. This reflects the intuition that small deviations in large boxes matter less than in small boxes — a 10-pixel error on a 200-pixel box is less severe than on a 20-pixel box.

2. Weighted terms: \lambda_{\text{coord}} = 5 upweights localization loss, while \lambda_{\text{noobj}} = 0.5 downweights confidence loss for every nonresponsible box, including the other predictor in an occupied cell. This is necessary because most grid cells contain no object, and without the weighting, the gradient signal from empty cells would overwhelm the useful signal from cells containing objects.

3. Indicator functions: \mathbb{1}_{ij}^{\text{obj}} is 1 when the j-th box predictor in cell i is "responsible" for an object (i.e., has the highest IoU with the ground truth among the B predictors in that cell).

Non-Maximum Suppression

After the network produces 98 scored bounding boxes, NMS removes duplicates. Within each class, boxes are sorted by their class-specific score; a lower-scoring box is suppressed when its IoU with a retained box exceeds the threshold (0.5 in the demonstration). The paper reports a 2–3 mAP gain from NMS; 0.5 is a demonstration threshold, not a threshold specified in that paragraph.

Implementation

The grid, responsibility rule, and NMS now fit together as executable operations. Follow the original objective from labels to a gradient update, then inspect three concrete cases before comparing the paper’s benchmark results.

This offline PyTorch program implements the original detection mechanics with S = 7, B = 2, C = 20: 1470 outputs. The pinned author release config uses B = 3 and 1715 outputs. It also contains batch normalization and a local layer, so that release config is not an exact transcription of the paper architecture. No authenticated original pretrained checkpoint is included. All numbers below come from constructed prediction tensors, with no image model involved.

Download Python · Pinned requirements · Author’s release config

python -m pip install -r requirements-yolo.txt
python yolo_training.py --smoke --export inspection.json

After installing PyTorch, the program runs without network access or weights. The displayed code is generated from this exact executed download.

1. Put each object in the cell containing its center

Original paper

Enlarge figure

Math and tensor shapes

(r,c)=(\lfloor Sy\rfloor,\lfloor Sx\rfloor)

(t_x,t_y)=(Sx-c,Sy-r)

The target holds one object per cell, independent of the two candidate boxes. Width and height remain normalized to the whole image. This reference raises an error for cell collisions instead of silently discarding a label. Centers lie in [0, 1); an exact boundary belongs to the cell on its right or below.

Labels → target [N, 7, 7, 25]: occupancy + xywh + 20 classes

Python implementation

python

def encode_targets(labels):
    """Encode per-image (class, cx, cy, width, height) normalized labels."""
    if not labels:
        raise ValueError("At least one image is required, even if it is empty.")
    target = torch.zeros(len(labels), S, S, 5 + C)
    for n, objects in enumerate(labels):
        for label in objects:
            if len(label) != 5:
                raise ValueError(
                    "Each label needs class, cx, cy, width, height."
                )
            category, cx, cy, width, height = label
            if not all(torch.isfinite(torch.tensor(float(v))) for v in label):
                raise ValueError("Labels must be finite.")
            if category != int(category) or not 0 <= category < C:
                raise ValueError("Class must be an integer in [0, 20).")
            if not (
                0 <= cx < 1
                and 0 <= cy < 1
                and 0 < width <= 1
                and 0 < height <= 1
            ):
                raise ValueError("Centers must be in [0,1); sizes in (0,1].")
            row, column = int(cy * S), int(cx * S)
            if target[n, row, column, 0]:
                raise ValueError("YOLOv1 target supports one object per cell.")
            target[n, row, column, :5] = torch.tensor(
                [1, cx * S - column, cy * S - row, width, height]
            )
            target[n, row, column, 5 + int(category)] = 1
    return target

2. Decode cell offsets into image coordinates

Original paper

Enlarge figure

Math and tensor shapes

(x,y)=((c+t_x)/7,(r+t_y)/7)

b=(x-w/2,y-h/2,x+w/2,y+h/2)

This teaching tensor stores direct positive w and h. Darknet with sqrt=1 stores predicted square roots, so it squares those two outputs when decoding. The equation-level loss below takes square roots of direct sizes. Its gradients therefore differ from native output-parameter gradients. Assignment uses unclipped boxes; this example clips only for post-processing.

[N, 7, 7, 2, 4] → normalized xyxy with the same shape

Python implementation

python

def decode_boxes(local_boxes):
    """Convert [N, S, S, K, 4] local xy/image wh into image-normalized xyxy."""
    if (
        local_boxes.ndim != 5
        or tuple(local_boxes.shape[1:3]) != (S, S)
        or local_boxes.shape[-1] != 4
    ):
        raise ValueError("Boxes must have shape [N, 7, 7, K, 4].")
    if (
        not torch.isfinite(local_boxes).all()
        or (local_boxes[..., 2:] < 0).any()
    ):
        raise ValueError("Boxes require finite values and nonnegative sizes.")
    rows, columns = torch.meshgrid(
        torch.arange(S, device=local_boxes.device),
        torch.arange(S, device=local_boxes.device),
        indexing="ij",
    )
    grid = torch.stack((columns, rows), dim=-1)[None, :, :, None]
    center = (local_boxes[..., :2] + grid) / S
    half_size = local_boxes[..., 2:] / 2
    return torch.cat((center - half_size, center + half_size), dim=-1)

3. Select one responsible predictor by IoU

Original paper

Enlarge figure

Math and tensor shapes

\operatorname{IoU}(a,b)=|a\cap b|/|a\cup b|

j^*=\arg\max_j\operatorname{IoU}(b_j,g)

no_grad freezes assignment and the IoU used as the confidence target. There is one winner in an occupied cell and none in an empty cell. This reference chooses the first box for every tie, including two zero IoUs. Native Darknet chooses the smaller RMSE in the all-zero case; the paper does not specify a tie rule.

IoU [N, 7, 7, 2] → Boolean responsibility [N, 7, 7, 2]

Python implementation

python

def box_iou(first, second):
    """Compute broadcastable xyxy IoU; zero-area pairs have IoU zero."""
    intersection_size = (
        torch.minimum(first[..., 2:], second[..., 2:])
        - torch.maximum(first[..., :2], second[..., :2])
    ).clamp_min(0)
    intersection = intersection_size.prod(dim=-1)
    first_area = (first[..., 2:] - first[..., :2]).clamp_min(0).prod(-1)
    second_area = (second[..., 2:] - second[..., :2]).clamp_min(0).prod(-1)
    union = first_area + second_area - intersection
    return intersection / union.clamp_min(torch.finfo(first.dtype).tiny)


@torch.no_grad()
def assign_boxes(boxes, target):
    """Freeze both winner selection and the IoU confidence target."""
    predicted = decode_boxes(boxes[..., :4])
    truth = decode_boxes(target[..., 1:5].unsqueeze(-2))
    ious = box_iou(predicted, truth)
    # torch.argmax resolves every tie, including all-zero IoU, to index zero.
    winner = ious.argmax(dim=-1)
    occupied = target[..., 0].bool()
    responsible = functional.one_hot(winner, B).bool() & occupied[..., None]
    return responsible, ious

4. Regress only the responsible box

Original paper

Enlarge figure

Math and tensor shapes

L_{xy}=5\sum_{i,j}R_{ij}\lVert t_{ij}-\hat t_i\rVert^2

L_{wh}=5\sum_{i,j}R_{ij}\lVert s_{ij}^{1/2}-\hat s_i^{1/2}\rVert^2

Selecting the responsible rows leaves the other box with no coordinate gradient. The public loss rejects nonpositive predicted sizes before sqrt, making this direct-size domain explicit. A full network needs a suitable size parameterization; the offline step optimizes a constructed output tensor, not a CNN. An empty selection gives a zero coordinate loss.

R selected boxes [M, 4]; M is the occupied-cell count

Python implementation

python

def coordinate_loss(boxes, target, responsible):
    """Eq. 3: only the responsible predictor learns center and size."""
    predicted = boxes[..., :4][responsible]
    truth = target[..., None, 1:5].expand(-1, -1, -1, B, -1)[responsible]
    # Public loss validation rejects zero/negative predicted dimensions first.
    centers = (predicted[..., :2] - truth[..., :2]).square().sum()
    sizes = (predicted[..., 2:].sqrt() - truth[..., 2:].sqrt()).square().sum()
    return 5 * (centers + sizes)

5. Add confidence and occupied-cell class losses

Original paper

Enlarge figure

Math and tensor shapes

L_o=\sum R(C-\operatorname{stopgrad}(IoU))^2

L_n=0.5\sum(1-R)C^2

L_c=\sum_i O_i\lVert p_i-\hat p_i\rVert^2

The nonresponsible candidate in an occupied cell still learns confidence zero. Class regression occurs once per occupied cell, not once per candidate. Both classes and confidences use squared error on linear outputs, without a softmax or binary cross-entropy. The returned scalar follows Equation 3; it is not a reproduction of Darknet’s printed diagnostic cost.

Four scalar terms; sum per image, then mean over N images

Python implementation

python

def yolo_loss(prediction, target):
    """Return Eq. 3 terms, summed over cells/boxes/classes, averaged over N."""
    boxes, classes = validate_predictions(prediction)
    validate_target(target, prediction)
    responsible, ious = assign_boxes(boxes, target)
    confidence = boxes[..., 4]
    occupied = target[..., 0].bool()
    terms = {
        "coordinates": coordinate_loss(boxes, target, responsible),
        "object": (confidence[responsible] - ious[responsible]).square().sum(),
        "no_object": 0.5 * confidence[~responsible].square().sum(),
        "classes": (classes[occupied] - target[..., 5:][occupied])
        .square()
        .sum(),
    }
    terms = {name: value / prediction.shape[0] for name, value in terms.items()}
    return sum(terms.values()), terms

6. Multiply shared class scores by box confidence

Original paper

Enlarge figure

Math and tensor shapes

s_{ijc}=p_{ic}C_{ij}

Broadcasting reuses the same twenty class outputs for both boxes. With p(class 0) = 0.9 and confidences 0.6 and 0.8, the scores are 0.54 and 0.72. Linear outputs are not guaranteed to be calibrated probabilities. Corners are clipped to [0, 1] here before NMS; this is an explicit post-processing choice, not a claim about native clipping order.

Boxes [N, 98, 4]; scores [N, 98, 20]

Python implementation

python

def score_boxes(prediction):
    """Eq. 1: shared cell class scores times each box's confidence."""
    boxes, classes = validate_predictions(prediction)
    corners = decode_boxes(boxes[..., :4]).clamp(0, 1)
    scores = boxes[..., 4, None] * classes[..., None, :]
    # No softmax/sigmoid: the paper trains linear final outputs with SSE.
    return corners.reshape(-1, S * S * B, 4), scores.reshape(-1, S * S * B, C)

7. Suppress overlaps separately for each class

Original paper

Enlarge figure

Math and tensor shapes

\operatorname{keep}(b)\iff\operatorname{IoU}(b,b_{selected})\le\tau

A detection in one class cannot suppress another class. Score threshold 0.2 and IoU threshold 0.5 are demonstration choices. Stable sorting gives equal scores a reproducible first-index order. The function consumes one image at a time; normalized boxes can then be scaled by image width and height for display.

[98, 4] and [98, 20] → (box index, class index, score) records

Python implementation

python

def classwise_nms(corners, scores, score_threshold=0.2, iou_threshold=0.5):
    """Return (box index, class index, score), keeping classes independent."""
    if (
        corners.ndim != 2
        or corners.shape[-1] != 4
        or scores.shape != (corners.shape[0], C)
    ):
        raise ValueError("Expected boxes [K,4] and scores [K,20].")
    if not 0 <= score_threshold <= 1 or not 0 <= iou_threshold <= 1:
        raise ValueError("Thresholds must lie in [0,1].")
    if not torch.isfinite(corners).all() or not torch.isfinite(scores).all():
        raise ValueError("NMS inputs must be finite.")
    if (corners[:, 2:] < corners[:, :2]).any():
        raise ValueError("NMS corners must be ordered as x1,y1,x2,y2.")
    keep = []
    for category in range(C):
        active = torch.where(scores[:, category] > score_threshold)[0]
        order = scores[active, category].argsort(descending=True, stable=True)
        active = active[order]
        while active.numel():
            selected = int(active[0])
            keep.append((selected, category, float(scores[selected, category])))
            rest = active[1:]
            overlaps = box_iou(corners[selected], corners[rest])
            active = rest[overlaps <= iou_threshold]
    return keep

8. Check the objective with an actual gradient step

Original paper

Enlarge figure

Math and tensor shapes

P\leftarrow P-0.01\nabla_P L

The smoke command executes encoding, decoding, assignment, every loss term, scoring, NMS, and one SGD update. It records before/after loss and gradient norm. It does not implement the 24-layer backbone, ImageNet pretraining, VOC augmentation, or the detection training schedule, and makes no accuracy or throughput claim.

Learnable [1, 7, 7, 30] tensor → scalar loss → same-shaped gradient

Python implementation

python

def update_predictions(prediction, target):
    """One SGD step on a learnable output tensor, without a backbone."""
    output = torch.nn.Parameter(prediction.clone())
    optimizer = torch.optim.SGD([output], lr=0.01)
    optimizer.zero_grad(set_to_none=True)
    before, terms = yolo_loss(output, target)
    before.backward()
    gradient_norm = float(output.grad.norm())
    optimizer.step()
    after, _ = yolo_loss(output.detach(), target)
    return {
        "before": float(before.detach()),
        "after": float(after),
        "gradient_norm": gradient_norm,
        "terms": {name: float(value.detach()) for name, value in terms.items()},
    }

Inspect the executed loss

Saved CPU results from three constructed fixtures. Box indices are zero-based. Changing the case selects saved evidence; it does not run Python in your browser.

Prediction caseOne exact matchNeither box overlapsNo objects

Responsible box: 1. Candidate IoUs: 0.2500, 1.0000.

coordinates loss

0.000000

object loss

0.040000

no object loss

0.180000

classes loss

0.020000

Total before SGD

0.240000

Total after SGD

0.234042

Class 0 scores in cell (3, 3): 0.54, 0.72. NMS keeps 2 records at score > 0.2 and IoU > 0.5 suppression.

Box 1 matches the target exactly. Box 0 still incurs a no-object confidence penalty, even though its cell is occupied. Their mutual IoU is 0.25, so both survive this NMS threshold.

Download the executed trace and provenance

Key Results

On PASCAL VOC 2007, YOLO achieves 63.4 mAP at 45 FPS. For context, Fast R-CNN reaches 70.0 mAP but at only 0.5 FPS, and Faster R-CNN achieves 73.2 mAP at 7 FPS. YOLO trades roughly 10 points of mAP for a 6–90x speedup depending on the baseline.

When combined with Fast R-CNN as an ensemble, accuracy jumps to 75.0 mAP — a 3.2 point boost over the 71.8 mAP Fast R-CNN baseline used in the paper’s separate combination experiment. This gain comes because YOLO makes far fewer background false positives (due to its global reasoning), complementing R-CNN's stronger localization.

On generalization, YOLO significantly outperforms DPM and R-CNN when transferring from natural images to artwork (Picasso and People-Art datasets). The gap is substantial: YOLO's mAP drops much less when the domain shifts, indicating that it learns more generalizable object representations rather than overfitting to photographic textures.

Comparison with the R-CNN Family

The fundamental trade-off is clear: YOLO sacrifices localization accuracy (especially on small objects) for throughput. Two-stage detectors use region proposals to focus computation on likely object locations, achieving better per-box accuracy. YOLO compensates with global reasoning — it sees the full image context when making predictions, which reduces background confusion errors by nearly half compared to Fast R-CNN.

Critical Analysis

Strengths:

• Speed through architectural simplicity. One forward pass replaces multi-stage pipelines. The 45 FPS throughput enabled real-time detection applications that were previously impractical.
• Global reasoning. Because the network sees the entire image during prediction, it encodes contextual information that region-based methods miss. This reduces background false positives and improves domain transfer.
• Clean training. End-to-end optimization with a single loss function eliminates the need to separately tune proposal generation, feature extraction, and classification stages.

Limitations:

• Struggles with small objects. The coarse 7x7 grid limits spatial resolution. Each cell predicts only 2 boxes, so densely packed small objects (e.g., a flock of birds) are systematically missed.
• Grid constraints impose a hard ceiling. With S^2 \times B = 98 total predictions, the network cannot detect more than 98 objects. More critically, each cell can only detect one object class, so nearby objects of different classes that share a cell will lose one detection.
• Localization errors dominate. The paper's error analysis shows that YOLO's primary failure mode is imprecise bounding boxes, not misclassification. The sum-of-squared-errors loss treats localization errors equally across box sizes despite the square-root mitigation.
• Aspect ratio sensitivity. The network is trained on a fixed 448x448 input, and the FC layers bake in spatial assumptions. Objects with unusual aspect ratios or at uncommon scales receive less training signal.

The YOLO Lineage

YOLO spawned one of the most prolific lineages in object detection:

• YOLOv2 / YOLO9000 (Redmon & Farhadi, 2017) — introduced batch normalization, anchor boxes, multi-scale training, and WordTree for detecting 9000+ categories. Replaced FC layers with fully convolutional architecture.
• YOLOv3 (Redmon & Farhadi, 2018) — adopted feature pyramid networks for multi-scale detection, switched to logistic classifiers for multi-label prediction, and used Darknet-53 as the backbone.
• YOLOv4 (Bochkovskiy et al., 2020) — systematic integration of bag-of-freebies (data augmentation, label smoothing) and bag-of-specials (SPP, PAN) for optimized training.
• YOLOv5–v11 (Ultralytics, various authors, 2020–2024) — continued iterations on architecture, training recipes, and deployment optimization. Later versions adopted anchor-free detection and transformer-based attention mechanisms.

Each version addressed specific YOLOv1 limitations — anchor boxes for aspect ratio diversity, feature pyramids for small objects, fully convolutional architectures for variable input sizes — while preserving the core single-pass philosophy.

Impact and Legacy

YOLOv1 did not achieve the highest accuracy on any benchmark. Its lasting contribution was demonstrating that object detection could be reframed as a direct regression problem without sacrificing practical utility. This insight — that a single-stage detector could achieve real-time performance with acceptable accuracy — shifted the field's trajectory.

The paper directly motivated SSD (Liu et al., 2016), which combined YOLO's single-pass approach with multi-scale feature maps. RetinaNet (Lin et al., 2017) later showed that the accuracy gap between one-stage and two-stage detectors was primarily due to class imbalance during training, not architectural limitations, and closed it with focal loss. Modern anchor-free detectors like FCOS and CenterNet further simplified the YOLO paradigm by eliminating predefined anchor boxes altogether.

Beyond research, YOLO popularized real-time object detection in industry. Autonomous driving perception stacks, video surveillance systems, and edge-device inference pipelines all trace design lineage back to the principle that detection speed and accuracy need not be fundamentally at odds.

Related Reading

• Faster R-CNN — the two-stage detector that YOLO was benchmarked against, introducing Region Proposal Networks
• DETR — end-to-end object detection with transformers, eliminating NMS and anchors entirely
• Deep Residual Learning — ResNets that became standard backbones for later YOLO versions
• Vision Transformer — the ViT architecture that influenced modern detection backbones