Skip to main content

Semantic, Instance, and Panoptic Segmentation

Compare semantic, instance and panoptic label maps, inspect individual pixels, and understand dense predictions, mask IoU, Dice and boundary errors.

A bounding box encloses an object, including some background. Segmentation describes which pixels belong to a region. Before choosing an architecture, decide what the region labels must represent: a class, an individual object, or a complete scene containing both objects and background regions.

Change the question, keep the scene

The scene below contains two cars, one person, sky, and road. Switch the task and inspect a pixel. Both cars share a semantic class, but they need different instance identities if the goal is to distinguish individual vehicles.

Three questions about the same pixels
Segmentation task

Teal: cars share one class. Violet: person. Sky blue: sky; gray: road.

9
6
Class
car
Instance ID
Not represented

Authored label map, not model output. Semantic labels merge the two cars into one class. Instance labels separate objects; panoptic labels also cover sky and road. The sliders provide a keyboard-accessible pixel probe.

Three output contracts

Semantic segmentation assigns a class to each labeled pixel. Both cars receive the car class. The output does not say whether adjacent car pixels belong to one car or several cars.

Instance segmentation returns a separate mask for each object instance, usually with a class and confidence score. It distinguishes car 1 from car 2, but it need not label sky and road. Some instance representations allow overlapping predicted masks before any conflict resolution.

Panoptic segmentation combines semantic coverage with instance identities for countable objects, often called “things.” Regions such as sky and road are “stuff” classes without object-instance identities. A panoptic map assigns a single class/instance interpretation to each evaluated pixel. The original paper defines the unified task and its panoptic quality metric. Panoptic Segmentation

An instance ID is local identity, not a class. Car 2 is still a car, and ID 2 in another image need not represent the same physical vehicle. Tracking across frames introduces an additional identity problem.

Predict densely without discarding all spatial detail

A classifier compresses spatial evidence into an image-level output. A segmentation network retains or reconstructs spatial predictions. For a semantic task with CC classes, one common output is a tensor of class logits shaped (C, H, W), followed by a class choice at each pixel.

Backbone features may be coarser than the desired label grid. Upsampling restores output resolution, while finer features can help recover boundaries. Fully Convolutional Networks demonstrated dense prediction by adapting classification networks and combining coarse semantic features with finer appearance information. FCN paper

Increasing output resolution alone cannot guarantee accurate boundaries. If a thin structure disappeared during downsampling, an interpolation step has no direct evidence of its exact shape. This connects segmentation to receptive fields, dilated convolutions, and skip connections.

Evaluate the mask, not its enclosing box

For a binary predicted mask PP and target mask TT, overlap can be measured with mask IoU or Dice:

IoU=∣P∩T∣∣P∪T∣,Dice=2∣P∩T∣∣P∣+∣T∣IoU = \frac{|P\cap T|}{|P\cup T|},\qquad Dice = \frac{2|P\cap T|}{|P|+|T|}

Suppose the prediction covers 12 pixels, the target covers 10, and 8 overlap. The union is 14, giving IoU 8/14 ≈ 0.571; Dice is 16/22 ≈ 0.727. These scores use pixels in the masks, not areas of enclosing rectangles. State how empty masks, ignored pixels, and class averaging are handled.

Pixel accuracy alone can conceal failure when background dominates. Predicting all background in an image with a tiny foreground object can score highly while missing that object completely. Inspect per-class overlap and boundaries, and use the evaluation protocol appropriate to semantic, instance, or panoptic outputs.

Finally, keep labels aligned during augmentation. A geometric transform must move the mask with the image, while its interpolation must preserve discrete label IDs.

Continue with dense prediction and upsampling to follow the decoder that turns coarse features into an aligned output grid, and see why enlarging that grid alone cannot restore missing detail.

If you found this explanation helpful, consider sharing it with others.

Mastodon