Segmentation asks for a decision at each pixel. The features that support that decision may live on a much smaller grid. A decoder connects these two resolutions: it enlarges spatial representations, combines useful detail, and produces predictions aligned with the image.
Making an output grid larger is only part of this job. A larger grid can contain smoother versions of exactly the same incomplete evidence.
From image features to a dense output
For a semantic segmentation task with classes, a common output is a tensor of logits shaped (N, C, H, W). At each spatial position, its numbers compete to determine the class. Other dense tasks use different channels: depth might predict one continuous value, while optical flow predicts two displacement components.
Repeated strided operations let an encoder gather context efficiently. In an architecture with output stride 16, a 256 × 256 input can produce a 16 × 16 feature grid. Here, output stride describes the spacing of feature locations relative to the input; it does not say that each feature observes only a 16 × 16 patch. Its receptive field may be much larger.
A decoder can first predict coarse class logits and resize them, or enlarge and fuse feature maps before predicting logits. Fully Convolutional Networks established an influential dense prediction design combining deep, coarse semantic information with shallower, finer information. FCN paper
Resampling changes the grid
Nearest-neighbor upsampling copies a source value. Bilinear upsampling takes a weighted combination of neighboring values. Both are fixed rules: their interpolation weights come from sample locations, rather than from training.
The explorer uses the same half-pixel convention as the resizing concept. An output column maps to input coordinate:
The same calculation applies to rows, with samples clamped at the boundary. For 2 → 4 enlargement, the input coordinates are −0.25, 0.25, 0.75, 1.25. Bilinear interpolation between margins −2 and +2 therefore gives −2, −1, +1, +2. Nearest selects the closest input center; this toy rounds an exact midpoint toward the higher index.
Compare Nearest and Bilinear, then inspect Transposed conv. Finally select Skip fusion and move its gain from 0 to 1: the added fine feature changes two pixels that interpolation leaves as background.
Coarse margins · 2 × 2
Output margins · F: foreground, B: background. Select a cell to inspect its arithmetic.
Pixel (1, 0)
Input coordinate (row, col): (0.25, -0.25), clamped at the edges.
(-2 × 0.75) + (-2 × 0.25) = -2
Authored arithmetic, not a trained model. Both methods reuse the same coarse evidence. Bilinear smooths the margins, but neither method recovers a thin structure absent from those margins.
The displayed quantity is a logit margin: foreground logit minus background logit. Its sign determines the two-class decision, with a stated tie rule. It is not a probability. Interpolating this margin is equivalent to interpolating both logits and then subtracting, because the interpolation is linear.
The thin foreground feature is deliberately absent from the coarse margins. Bilinear interpolation can make their transition smoother, but cannot identify the missing feature's location. A learned decoder may infer plausible detail from learned patterns; a skip path can also supply fine evidence retained by the encoder. Neither makes the result automatically correct.
A transposed convolution learns how values spread
A transposed convolution multiplies each input value by a kernel and accumulates its contributions on an output grid. The kernel weights can be learned. The explorer fixes them by hand to expose the arithmetic: kernel size 2 and stride 2 write separate 2 × 2 blocks without overlap. Other configurations overlap, so an output value may sum contributions from several input positions.
For one axis, its output size is:
Here is stride, padding, dilation, kernel size, and output padding. With , the output has length 4. Output padding resolves shape ambiguity from strided operations; it does not append a border of zero values. A transposed convolution is not an inverse that restores values discarded by an encoder. PyTorch ConvTranspose2d
Learned spatial weights and overlap can produce spatially varying responses even for simple inputs. Check output patterns as well as dimensions. Another decoder design resizes first and then applies ordinary convolutions, separating the fixed sampling rule from the learned refinement.
Fine features provide another path to the prediction
An encoder skip connection carries an earlier, finer feature map into a decoder stage. It complements coarse context with local evidence. U-Net combines an expanding path with corresponding encoder features; its original architecture crops the encoder features before concatenation because unpadded convolutions shrink their spatial dimensions. U-Net paper, network architecture
Two common fusion operations have different contracts:
| Fusion | Required alignment | Result |
|---|---|---|
| Concatenate along channels | Same batch and spatial grid; channel counts may differ | Keeps both channel groups, then a learned layer can mix them |
| Add element by element | Same batch, spatial grid, and channel count | Combines corresponding channels; a projection can first make channels compatible |
For example, (N, 64, 32, 32) decoder features and (N, 32, 32, 32) skip features concatenate into (N, 96, 32, 32). They cannot be added directly. A learned 1 × 1 projection could map the skip features to 64 channels before addition. Shape compatibility is necessary; the features must also refer to the same image locations.
The explorer's skip grid represents an already projected fine encoder feature. Adding it to the enlarged margin is an authored scalar fusion example. Real decoders commonly fuse many feature channels and predict class logits afterward.
Match coordinates as well as shapes
Two arrays can have the same height and width while representing shifted locations. Padding, cropping, odd input dimensions, and resizing conventions all affect alignment. Track how the output grid maps back to the input, including any preprocessing crop or letterbox padding.
For bilinear resizing, align_corners=False uses the half-pixel geometry above. With align_corners=True and output length greater than one, the endpoint centers instead align through . For 2 → 4, the interior coordinates become 1/3 and 2/3 instead of 1/4 and 3/4. Keep this choice consistent across training and inference, and specify the destination spatial size when it is known. Nearest modes have their own sampling conventions. PyTorch interpolate
The following teaching decoder explicitly resizes to the skip grid before fusion, then to the requested output grid. It assumes the two feature grids already have compatible coordinate origins and extents; resizing alone does not establish that geometry. fuse and classifier are learned modules with matching input channel counts.
import torch
import torch.nn.functional as F
def decode_logits(coarse, skip, fuse, classifier, output_hw):
upsampled = F.interpolate(
coarse, size=skip.shape[-2:], mode="bilinear", align_corners=False
)
features = fuse(torch.cat((upsampled, skip), dim=1))
logits = classifier(features)
return F.interpolate(
logits, size=output_hw, mode="bilinear", align_corners=False
)
# Choose a class only after logits reach the desired grid.
# labels = decode_logits(...).argmax(dim=1)Resize logits before choosing classes
An argmax discards all scores except the winning class. Resizing coarse logits and then taking argmax can therefore differ from taking coarse argmax and enlarging its labels.
Consider neighboring two-class margins −3 and +1. At a position three quarters of the way from the first center to the second, interpolation gives 0.25 × (−3) + 0.75 × 1 = 0. With the explorer's tie rule, this selects background. Nearest resizing of the already chosen labels selects the second center's foreground label. The ordering changes the result because the label map has lost the confidence margins.
Once the data are discrete class IDs, preserve those IDs with a label-safe operation such as nearest-neighbor. Bilinear interpolation of IDs has no class meaning: an average between background 0 and car 2 could create 1, the ID of another class.
The decoder's final contract is an aligned prediction grid with well-defined channel meaning. Segmentation evaluation can then measure its overlap and boundary errors. More pixels alone establish neither better localization nor better predictions.
Related concepts
Compare semantic, instance and panoptic label maps, inspect individual pixels, and understand dense predictions, mask IoU, Dice and boundary errors.
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
Follow a 3D world point through camera extrinsics, depth division and pixel intrinsics, then see why one image point cannot determine its depth.
