A bounding box encloses an object, including some background. Segmentation describes which pixels belong to a region. Before choosing an architecture, decide what the region labels must represent: a class, an individual object, or a complete scene containing both objects and background regions.
Change the question, keep the scene
The scene below contains two cars, one person, sky, and road. Switch the task and inspect a pixel. Both cars share a semantic class, but they need different instance identities if the goal is to distinguish individual vehicles.
Teal: cars share one class. Violet: person. Sky blue: sky; gray: road.
- Class
- car
- Instance ID
- Not represented
Authored label map, not model output. Semantic labels merge the two cars into one class. Instance labels separate objects; panoptic labels also cover sky and road. The sliders provide a keyboard-accessible pixel probe.
Three output contracts
Semantic segmentation assigns a class to each labeled pixel. Both cars receive the car class. The output does not say whether adjacent car pixels belong to one car or several cars.
Instance segmentation returns a separate mask for each object instance, usually with a class and confidence score. It distinguishes car 1 from car 2, but it need not label sky and road. Some instance representations allow overlapping predicted masks before any conflict resolution.
Panoptic segmentation combines semantic coverage with instance identities for countable objects, often called “things.” Regions such as sky and road are “stuff” classes without object-instance identities. A panoptic map assigns a single class/instance interpretation to each evaluated pixel. The original paper defines the unified task and its panoptic quality metric. Panoptic Segmentation
An instance ID is local identity, not a class. Car 2 is still a car, and ID 2 in another image need not represent the same physical vehicle. Tracking across frames introduces an additional identity problem.
Predict densely without discarding all spatial detail
A classifier compresses spatial evidence into an image-level output. A segmentation network retains or reconstructs spatial predictions. For a semantic task with classes, one common output is a tensor of class logits shaped (C, H, W), followed by a class choice at each pixel.
Backbone features may be coarser than the desired label grid. Upsampling restores output resolution, while finer features can help recover boundaries. Fully Convolutional Networks demonstrated dense prediction by adapting classification networks and combining coarse semantic features with finer appearance information. FCN paper
Increasing output resolution alone cannot guarantee accurate boundaries. If a thin structure disappeared during downsampling, an interpolation step has no direct evidence of its exact shape. This connects segmentation to receptive fields, dilated convolutions, and skip connections.
Evaluate the mask, not its enclosing box
For a binary predicted mask and target mask , overlap can be measured with mask IoU or Dice:
Suppose the prediction covers 12 pixels, the target covers 10, and 8 overlap. The union is 14, giving IoU 8/14 ≈ 0.571; Dice is 16/22 ≈ 0.727. These scores use pixels in the masks, not areas of enclosing rectangles. State how empty masks, ignored pixels, and class averaging are handled.
Pixel accuracy alone can conceal failure when background dominates. Predicting all background in an image with a tiny foreground object can score highly while missing that object completely. Inspect per-class overlap and boundaries, and use the evaluation protocol appropriate to semantic, instance, or panoptic outputs.
Finally, keep labels aligned during augmentation. A geometric transform must move the mask with the image, while its interpolation must preserve discrete label IDs.
Continue with dense prediction and upsampling to follow the decoder that turns coarse features into an aligned output grid, and see why enlarging that grid alone cannot restore missing detail.
Related concepts
Follow coarse feature maps into aligned pixel predictions: nearest and bilinear interpolation, transposed convolution, output shapes, skip fusion, and resizing logits before selecting labels.
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
Follow a 3D world point through camera extrinsics, depth division and pixel intrinsics, then see why one image point cannot determine its depth.
