Skip to main content

Image Augmentation and Label Consistency

Transform images and annotations together, inspect crop and flip geometry, and choose augmentation assumptions that preserve the meaning of the task.

Augmentation creates training variations from existing examples. It encodes an assumption: after a transformation, the input should still have a valid, known target. That target may remain unchanged, as with some image-level labels, or it may need the same geometric transformation as the image.

Follow an object through a transform

Increase the crop amount below, then flip the result. The image and boxes follow the same crop, resize, and flip. Notice how clipping changes partially visible boxes and how stretching the cropped region changes object shape.

Transform the pixels and the labels together
Horizontal flip
0 px
Original · 320 × 200
Crop, resize to 320 × 200, then flip
car-a
30, 100, 120, 160
car-b
190, 100, 280, 160
person-a
140, 55, 170, 155

Authored geometry example. Boxes are clipped to the crop and transformed with the scene. This deliberately stretches a non-square crop back to the original size; watch how the objects change shape.

Sample geometry once, apply it everywhere

Suppose a crop begins at (left, top) and is resized by factors sxs_x and sys_y. A retained point maps to:

x′=(x−left)sx,y′=(y−top)syx'=(x-left)s_x,\qquad y'=(y-top)s_y

Apply this mapping to the image coordinates, boxes, keypoints, and masks. Sampling a separate random crop for each would produce inconsistent training pairs. A crop entirely outside an object's box removes that object; a partial crop needs a policy for clipping, visibility, and whether the remaining fragment is still a valid example.

For the starting left-car box (30, 100, 120, 160), a 20-pixel crop from every edge leaves a 280 × 160 canvas. Resizing it back to 320 × 200 uses scales 320/280 and 200/160. The transformed box is approximately (11.4, 100, 114.3, 175) before any flip. The unequal scales explain the visible distortion.

Preserve each target's meaning

An RGB image can use bilinear interpolation. A class-index mask needs label-preserving interpolation such as nearest-neighbor, since averaging IDs can invent another class. Keypoints need both coordinate changes and visibility updates; a horizontal flip may also require exchanging semantic labels such as left eye and right eye.

Photometric transforms, such as changing brightness, usually leave geometry unchanged. Their validity still depends on the task. A color change may preserve a generic vehicle label while altering evidence essential to a traffic-light classification task. A horizontal flip may be reasonable for some object categories but change the meaning of written text.

Torchvision's transforms v2 API can apply a shared random transformation to typed images, boxes, and masks. Its detection/segmentation example also shows sanitizing degenerate boxes while keeping associated targets aligned. Torchvision joint transforms

python
# examples: deterministic flip and resize for an image and its boxes
import torch
from torchvision import tv_tensors
from torchvision.transforms import v2

image = tv_tensors.Image(torch.zeros(3, 200, 320, dtype=torch.uint8))
boxes = tv_tensors.BoundingBoxes(
    [[30, 100, 120, 160]], format='XYXY', canvas_size=(200, 320)
)
transform = v2.Compose([
    v2.RandomHorizontalFlip(p=1.0),
    v2.Resize((100, 160)),
])
image, boxes = transform(image, boxes)
# Box: (100, 50, 145, 80) on the resized canvas.

Keep evaluation independent

Split the original data before generating augmented copies. Near-duplicate frames from the same video, camera session, or subject can otherwise leak across training and evaluation. Where deployment requires generalization across such groups, keep the relevant groups separate in the split.

Use a fixed, documented evaluation preprocessing pipeline. Test-time augmentation is a separate inference method that combines multiple transformed predictions and needs its own evaluation and compute accounting.

Start with transformations justified by likely input variation, inspect image-target pairs, and compare held-out results under the same protocol. More extreme augmentation is not automatically better. If failures cluster on small objects or cropped instances, review the target transformations and sampling policy before attributing the problem to the model architecture.

If you found this explanation helpful, consider sharing it with others.

Mastodon