Skip to main content

Dense Prediction and Upsampling

Follow coarse feature maps into aligned pixel predictions: nearest and bilinear interpolation, transposed convolution, output shapes, skip fusion, and resizing logits before selecting labels.

Segmentation asks for a decision at each pixel. The features that support that decision may live on a much smaller grid. A decoder connects these two resolutions: it enlarges spatial representations, combines useful detail, and produces predictions aligned with the image.

Making an output grid larger is only part of this job. A larger grid can contain smoother versions of exactly the same incomplete evidence.

From image features to a dense output

For a semantic segmentation task with CC classes, a common output is a tensor of logits shaped (N, C, H, W). At each spatial position, its CC numbers compete to determine the class. Other dense tasks use different channels: depth might predict one continuous value, while optical flow predicts two displacement components.

Repeated strided operations let an encoder gather context efficiently. In an architecture with output stride 16, a 256 × 256 input can produce a 16 × 16 feature grid. Here, output stride describes the spacing of feature locations relative to the input; it does not say that each feature observes only a 16 × 16 patch. Its receptive field may be much larger.

A decoder can first predict coarse class logits and resize them, or enlarge and fuse feature maps before predicting logits. Fully Convolutional Networks established an influential dense prediction design combining deep, coarse semantic information with shallower, finer information. FCN paper

Resampling changes the grid

Nearest-neighbor upsampling copies a source value. Bilinear upsampling takes a weighted combination of neighboring values. Both are fixed rules: their interpolation weights come from sample locations, rather than from training.

The explorer uses the same half-pixel convention as the resizing concept. An output column xox_o maps to input coordinate:

xi=(xo+0.5)WiWo−0.5x_i = (x_o + 0.5)\frac{W_i}{W_o} - 0.5

The same calculation applies to rows, with samples clamped at the boundary. For 2 → 4 enlargement, the input coordinates are −0.25, 0.25, 0.75, 1.25. Bilinear interpolation between margins −2 and +2 therefore gives −2, −1, +1, +2. Nearest selects the closest input center; this toy rounds an exact midpoint toward the higher index.

Compare Nearest and Bilinear, then inspect Transposed conv. Finally select Skip fusion and move its gain from 0 to 1: the added fine feature changes two pixels that interpolation leaves as background.

What supplies the finer prediction?
Decoder

Coarse margins · 2 × 2

-2
+2
-2
+2

Output margins · F: foreground, B: background. Select a cell to inspect its arithmetic.

Pixel (1, 0)

Input coordinate (row, col): (0.25, -0.25), clamped at the edges.

(-2 × 0.75) + (-2 × 0.25) = -2

Authored arithmetic, not a trained model. Both methods reuse the same coarse evidence. Bilinear smooths the margins, but neither method recovers a thin structure absent from those margins.

The displayed quantity is a logit margin: foreground logit minus background logit. Its sign determines the two-class decision, with a stated tie rule. It is not a probability. Interpolating this margin is equivalent to interpolating both logits and then subtracting, because the interpolation is linear.

The thin foreground feature is deliberately absent from the coarse margins. Bilinear interpolation can make their transition smoother, but cannot identify the missing feature's location. A learned decoder may infer plausible detail from learned patterns; a skip path can also supply fine evidence retained by the encoder. Neither makes the result automatically correct.

A transposed convolution learns how values spread

A transposed convolution multiplies each input value by a kernel and accumulates its contributions on an output grid. The kernel weights can be learned. The explorer fixes them by hand to expose the arithmetic: kernel size 2 and stride 2 write separate 2 × 2 blocks without overlap. Other configurations overlap, so an output value may sum contributions from several input positions.

For one axis, its output size is:

Lo=(Li−1)s−2p+d(k−1)+o+1L_o = (L_i - 1)s - 2p + d(k - 1) + o + 1

Here ss is stride, pp padding, dd dilation, kk kernel size, and oo output padding. With Li=2,s=2,k=2,p=o=0,d=1L_i=2, s=2, k=2, p=o=0, d=1, the output has length 4. Output padding resolves shape ambiguity from strided operations; it does not append a border of zero values. A transposed convolution is not an inverse that restores values discarded by an encoder. PyTorch ConvTranspose2d

Learned spatial weights and overlap can produce spatially varying responses even for simple inputs. Check output patterns as well as dimensions. Another decoder design resizes first and then applies ordinary convolutions, separating the fixed sampling rule from the learned refinement.

Fine features provide another path to the prediction

An encoder skip connection carries an earlier, finer feature map into a decoder stage. It complements coarse context with local evidence. U-Net combines an expanding path with corresponding encoder features; its original architecture crops the encoder features before concatenation because unpadded convolutions shrink their spatial dimensions. U-Net paper, network architecture

Two common fusion operations have different contracts:

FusionRequired alignmentResult
Concatenate along channelsSame batch and spatial grid; channel counts may differKeeps both channel groups, then a learned layer can mix them
Add element by elementSame batch, spatial grid, and channel countCombines corresponding channels; a projection can first make channels compatible

For example, (N, 64, 32, 32) decoder features and (N, 32, 32, 32) skip features concatenate into (N, 96, 32, 32). They cannot be added directly. A learned 1 × 1 projection could map the skip features to 64 channels before addition. Shape compatibility is necessary; the features must also refer to the same image locations.

The explorer's skip grid represents an already projected fine encoder feature. Adding it to the enlarged margin is an authored scalar fusion example. Real decoders commonly fuse many feature channels and predict class logits afterward.

Match coordinates as well as shapes

Two arrays can have the same height and width while representing shifted locations. Padding, cropping, odd input dimensions, and resizing conventions all affect alignment. Track how the output grid maps back to the input, including any preprocessing crop or letterbox padding.

For bilinear resizing, align_corners=False uses the half-pixel geometry above. With align_corners=True and output length greater than one, the endpoint centers instead align through xi=xo(Wi−1)/(Wo−1)x_i=x_o(W_i-1)/(W_o-1). For 2 → 4, the interior coordinates become 1/3 and 2/3 instead of 1/4 and 3/4. Keep this choice consistent across training and inference, and specify the destination spatial size when it is known. Nearest modes have their own sampling conventions. PyTorch interpolate

The following teaching decoder explicitly resizes to the skip grid before fusion, then to the requested output grid. It assumes the two feature grids already have compatible coordinate origins and extents; resizing alone does not establish that geometry. fuse and classifier are learned modules with matching input channel counts.

Decoder with explicit destination sizes
import torch
import torch.nn.functional as F


def decode_logits(coarse, skip, fuse, classifier, output_hw):
    upsampled = F.interpolate(
        coarse, size=skip.shape[-2:], mode="bilinear", align_corners=False
    )
    features = fuse(torch.cat((upsampled, skip), dim=1))
    logits = classifier(features)
    return F.interpolate(
        logits, size=output_hw, mode="bilinear", align_corners=False
    )


# Choose a class only after logits reach the desired grid.
# labels = decode_logits(...).argmax(dim=1)

Resize logits before choosing classes

An argmax discards all scores except the winning class. Resizing coarse logits and then taking argmax can therefore differ from taking coarse argmax and enlarging its labels.

Consider neighboring two-class margins −3 and +1. At a position three quarters of the way from the first center to the second, interpolation gives 0.25 × (−3) + 0.75 × 1 = 0. With the explorer's tie rule, this selects background. Nearest resizing of the already chosen labels selects the second center's foreground label. The ordering changes the result because the label map has lost the confidence margins.

Once the data are discrete class IDs, preserve those IDs with a label-safe operation such as nearest-neighbor. Bilinear interpolation of IDs has no class meaning: an average between background 0 and car 2 could create 1, the ID of another class.

The decoder's final contract is an aligned prediction grid with well-defined channel meaning. Segmentation evaluation can then measure its overlap and boundary errors. More pixels alone establish neither better localization nor better predictions.

If you found this explanation helpful, consider sharing it with others.

Mastodon