Augmentation creates training variations from existing examples. It encodes an assumption: after a transformation, the input should still have a valid, known target. That target may remain unchanged, as with some image-level labels, or it may need the same geometric transformation as the image.
Follow an object through a transform
Increase the crop amount below, then flip the result. The image and boxes follow the same crop, resize, and flip. Notice how clipping changes partially visible boxes and how stretching the cropped region changes object shape.
- car-a
- 30, 100, 120, 160
- car-b
- 190, 100, 280, 160
- person-a
- 140, 55, 170, 155
Authored geometry example. Boxes are clipped to the crop and transformed with the scene. This deliberately stretches a non-square crop back to the original size; watch how the objects change shape.
Sample geometry once, apply it everywhere
Suppose a crop begins at (left, top) and is resized by factors and . A retained point maps to:
Apply this mapping to the image coordinates, boxes, keypoints, and masks. Sampling a separate random crop for each would produce inconsistent training pairs. A crop entirely outside an object's box removes that object; a partial crop needs a policy for clipping, visibility, and whether the remaining fragment is still a valid example.
For the starting left-car box (30, 100, 120, 160), a 20-pixel crop from every edge leaves a 280 × 160 canvas. Resizing it back to 320 × 200 uses scales 320/280 and 200/160. The transformed box is approximately (11.4, 100, 114.3, 175) before any flip. The unequal scales explain the visible distortion.
Preserve each target's meaning
An RGB image can use bilinear interpolation. A class-index mask needs label-preserving interpolation such as nearest-neighbor, since averaging IDs can invent another class. Keypoints need both coordinate changes and visibility updates; a horizontal flip may also require exchanging semantic labels such as left eye and right eye.
Photometric transforms, such as changing brightness, usually leave geometry unchanged. Their validity still depends on the task. A color change may preserve a generic vehicle label while altering evidence essential to a traffic-light classification task. A horizontal flip may be reasonable for some object categories but change the meaning of written text.
Torchvision's transforms v2 API can apply a shared random transformation to typed images, boxes, and masks. Its detection/segmentation example also shows sanitizing degenerate boxes while keeping associated targets aligned. Torchvision joint transforms
# examples: deterministic flip and resize for an image and its boxes
import torch
from torchvision import tv_tensors
from torchvision.transforms import v2
image = tv_tensors.Image(torch.zeros(3, 200, 320, dtype=torch.uint8))
boxes = tv_tensors.BoundingBoxes(
[[30, 100, 120, 160]], format='XYXY', canvas_size=(200, 320)
)
transform = v2.Compose([
v2.RandomHorizontalFlip(p=1.0),
v2.Resize((100, 160)),
])
image, boxes = transform(image, boxes)
# Box: (100, 50, 145, 80) on the resized canvas.
Keep evaluation independent
Split the original data before generating augmented copies. Near-duplicate frames from the same video, camera session, or subject can otherwise leak across training and evaluation. Where deployment requires generalization across such groups, keep the relevant groups separate in the split.
Use a fixed, documented evaluation preprocessing pipeline. Test-time augmentation is a separate inference method that combines multiple transformed predictions and needs its own evaluation and compute accounting.
Start with transformations justified by likely input variation, inspect image-target pairs, and compare held-out results under the same protocol. More extreme augmentation is not automatically better. If failures cluster on small objects or cropped instances, review the target transformations and sampling policy before attributing the problem to the model architecture.
Related concepts
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
Compare nearest-neighbor and bilinear image resizing, see aliasing when downsampling, and understand aspect ratio, letterboxing and mask interpolation.
Calculate box intersection and union, explore how position and size change IoU, and distinguish overlap used for evaluation, NMS and localization losses.
Understand object detection from image features to boxes, classes and scores. Compare one-stage and two-stage pipelines, training and duplicate handling.
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
