Skip to main content

U-Net: Convolutional Networks for Biomedical Image Segmentation

Reconstruct the original 572-to-388 valid-convolution U-Net, crop-and-concatenate skips, overlap tiling and boundary-weighted training, with executable geometry.

TL;DR

  • U-Net combines coarse context with saved high-resolution features through an expanding path with crop-and-concatenate skips.
  • The original unpadded architecture maps 572×572 inputs to 388×388 predictions. Cropping, the input halo and overlap tiling are part of its spatial contract.
  • Boundary-weighted training emphasizes gaps between cells, while paired augmentation supplies meaningful variation. The original two-class output does not directly assign instance identities.
  • Published biomedical results support the reported recipe and evaluation protocols. The executable reference checks geometry and weights, while padded modern U-Nets should be identified as variants.

Why boundaries need context and spatial detail

A cell boundary is a localization problem. Recognizing that a region contains cells does not say exactly where one cell ends and its neighbor begins. Pooling helps a convolutional network collect context, but it also makes a small displacement harder to recover. U-Net makes the return to spatial detail part of the architecture: an expanding path repeatedly combines coarse context with features saved at matching encoder scales.

This review follows the May 2015 original preprint, published at MICCAI 2015. The review date above describes this explanation. All interactive values are analytically calculated teaching examples. They are neither checkpoint outputs nor locally measured segmentation scores. The original PDF excerpts below are attributed source images. The companion Python program checks geometry and loss weights without pretending to train the network.

The prediction is an image, not a single label

Suppose a microscope tile contains two touching cells. A classifier could recognize the tile correctly while being useless for measuring either cell. A dense predictor instead returns a class distribution at each output location. Neighboring outputs can share most of their computation, so a fully convolutional pass avoids independently evaluating a large overlapping patch for every pixel.

The difficult part is reconciling two useful representations. Fine features retain changes in intensity, edges and texture. Deep features gather wider context and can distinguish a plausible cell boundary from an internal texture edge. A decoder that sees only the deepest map must infer spatial detail after repeated downsampling. A decoder that also receives earlier maps can use context to interpret the detail that survived there.

Think of this as a communication problem across resolutions. The low-resolution path answers what kind of structure is present and approximately where. The high-resolution path supplies evidence about its precise placement. Neither path is guaranteed to contain a correct boundary; both are learned features. The design simply gives later convolutions access to both kinds of information.

Rebuild the original geometry

The name U-Net now covers many architectures. To understand the 2015 paper, use its actual geometry: two unpadded 3×3 convolutions at each resolution, 2×2 max pooling on the way down, and 2×2 up-convolutions on the way up. A stride-one valid 3×3 convolution removes one pixel from each edge. A pair therefore changes an axis of length n into n−4. Original paper, Figure 1 and §2

Where did the border pixels go?

Analytically calculated original U-Net shapes. Every pair of unpadded 3×3 convolutions removes four pixels per axis.

Contracting path

Expanding path

568² × 64

two valid convolutions → pool ÷2

388² × 64

crop skip by 88 per side → concatenate

280² × 128

two valid convolutions → pool ÷2

196² × 128

crop skip by 40 per side → concatenate

136² × 256

two valid convolutions → pool ÷2

100² × 256

crop skip by 16 per side → concatenate

64² × 512

two valid convolutions → pool ÷2

52² × 512

crop skip by 4 per side → concatenate

Bottleneck: 28² × 1024

572 input → 388 output. Each output tile needs a 92-pixel input halo on every side.

Manual exploration only. This is shape arithmetic, not an inference run. The final 1×1 convolution produces two original-paper class logits per output pixel.

For the illustrated 572×572 input, the saved encoder widths are 568, 280, 136 and 64. Pooling the last gives 32; the bottleneck convolutions reduce it to 28. The expanding path then reaches widths 52, 100, 196 and finally 388 after its convolution pairs. The last 1×1 projection changes channels to the original two classes without changing spatial size.

These numbers explain why a modern “same padding” implementation does not reproduce Figure 1 simply by choosing the same channel widths. Padding changes which locations exist, what information is supplied at boundaries, and whether skip tensors need cropping. Batch normalization, residual blocks, bilinear resize decoders and multiclass heads may be useful variants, but they should be named as variants.

A compatible larger input adds the same amount to the output in this fixed-depth construction. The explorer uses input increments of sixteen so all four pooling operations divide exactly. An arbitrary odd input is not silently rounded to fit the diagram. A real implementation must specify its floor, padding and crop behavior when dimensions do not line up; a shape annotation is part of the algorithm, not just documentation.

The skip operation concatenates

At the first decoder merge, the bottleneck is up-convolved from 28 to 56 pixels. The corresponding encoder feature map is 64 pixels wide. Cropping four pixels from each edge makes it 56. Both maps have 512 channels, so concatenating them produces 1024 channels. Subsequent convolutions reduce the representation back to the desired decoder width.

Crop the skip, then concatenate channels

The skip carries spatial detail from the encoder. It is larger than the upsampled decoder map because valid convolutions have removed borders.

Encoder skip

64² × 512

remove 4 pixels on each edge

Decoder up-convolution

56² × 512

2× spatial expansion, half the channels

Channel concatenation

skipdecoder

56² × 1024

Merge 1: matching spatial size 56 × 56, then two valid convolutions produce 52 × 52 × 512.

The colored strips denote channel groups, not learned activation values. Concatenation preserves both groups; addition would require a different operation and keep only one group's width.

Addition would do something different. An elementwise sum requires corresponding channels to share one output slot. Concatenation gives a later learned filter separate access to the encoder and decoder channels. A filter can emphasize one, combine them, or ignore them. It does not need a predetermined alignment of semantic channel meanings between branches.

Cropping is also not a resizing operation. It retains the central values at their original spacing. Interpolating the entire 64-pixel skip down to 56 would move the feature samples relative to the decoder grid. The original valid-convolution construction instead discards the portion whose receptive-field support does not match the decoder's current prediction region.

There is a useful debugging invariant here: after cropping, the two tensors must agree in height and width, while their channel counts add. After the convolution pair, height and width each shrink by four again. If an implementation claims to reproduce the original network but never loses spatial extent, or adds skips without increasing channels, it implements a different construction.

Predict the center; supply the surroundings

The 572-to-388 relationship leaves a halo of 92 input pixels on each side of a predicted tile. The output does not simply represent every input pixel with a smaller image coordinate. It describes the valid central prediction region whose computation has sufficient surrounding context under this architecture.

For a large image, move the input window by the output width and stitch the central predictions. Neighboring input windows overlap even when their output regions meet without overlap. Near the outer image boundary, the paper supplies missing context by mirroring. Original paper, Figure 2

This overlap-tile strategy matters operationally. The model can process an image larger than GPU memory while retaining context around each output tile. It also creates costs that a simplistic “pixels per second” estimate can miss: overlapping inputs recompute some features, border handling must be consistent, and a stitching routine must place outputs at the correct offsets.

Mirroring is an assumption about the unobserved continuation of the image. It does not create new evidence beyond the image edge. Nor does tiling guarantee identical results to a hypothetical unlimited-memory computation under every architecture variant. Padding, normalization, global context modules and inference preprocessing can all change that equivalence. The original geometry gives an explicit contract that an implementation can test.

Weight the narrow gaps that matter

A foreground/background objective can tolerate a thin mistake between adjacent cells while changing a later connected-component interpretation dramatically. The paper therefore adds a spatial boundary term to a class-frequency weight. For a location x, let d₁ and d₂ be distances to the closest and second-closest distinct cell boundaries:

w(x)=w_c(x)+w_0\exp\!\left(-\frac{(d_1(x)+d_2(x))^2}{2\sigma^2}\right)

The reported settings use w₀=10 and approximately σ=5 pixels. Weighted pixelwise log loss makes selected errors contribute more strongly during optimization. The supplied class-frequency map and the geometric boundary term have different purposes. One compensates class prevalence; the other emphasizes locations near two instances. Original paper, §3, Equation 2

Why emphasize the gap between touching cells?

Calculated boundary weights along a one-dimensional gap. Two supplied nearest-instance distances satisfy d₁ + d₂ = gap width here; the class-frequency term is fixed to one.

4
5
50%
Cell Agap = 4 image pxCell B
Scale bar: 10 image pxDot: selected sample
d₁ · solid · nearest boundary A
2.00 image px
d₂ · dashed · other boundary B
2.00 image px
d₁ + d₂ = gap
4.00 image px

w = 1 + 10 exp(−4² / (2 × 5²)) = 8.2615

Narrower gaps receive more weight. With these parallel boundaries, moving the sample within one gap does not change d₁+d₂, so every location in that gap has the same weight.

This construction illustrates the weighting equation. It is not an observed cell image, learned mask, or proof that weighted loss alone separates instances. Distances in real training are measured to distinct instance boundaries.

The gap experiment deliberately uses parallel one-dimensional boundaries. At every point between them, d₁+d₂ equals the gap width. Consequently, all plotted weights inside that particular gap are equal. This is a useful counterexample to the tempting but incorrect claim that the formula must peak at the midpoint of every gap. In a two-dimensional shape, the distance geometry can be more varied.

The weight map is computed from training annotations. At inference time the model is not handed nearest-instance distances for the unknown answer. The training signal encourages separation, but the original two-class head remains a semantic foreground/background output. Recovering individual objects requires interpreting the predicted separating background; a two-class map does not inherently assign a unique identifier to every cell.

Weighting also cannot repair unreliable annotations automatically. A mislabeled narrow gap may be emphasized more strongly. Sigma has pixel units, so changing microscopy resolution or rescaling images changes its physical meaning unless preprocessing and weights are adjusted together. The toy program receives distances directly and leaves distance-transform construction and annotation policy to an actual data pipeline.

Augmentation is part of the small-data story

A network architecture alone does not explain why a system works with few annotated images. Augmentation specifies transformations under which the training problem should remain meaningful. If an image is deformed, its segmentation target must undergo the corresponding deformation. Applying independent random warps to the input and label would destroy the supervision.

The original work emphasizes elastic deformations alongside other augmentation. The broad lesson is that a biomedical training set can contain many pixels but still few independent specimens or acquisition conditions. Producing transformed views can expose useful geometric variation; it does not turn correlated samples into genuinely independent patients or experiments. Original paper, §3.1

A practical evaluation should therefore separate objects, images and acquisition groups carefully. Splitting overlapping tiles from one microscopy image between training and test sets can leak nearly identical structures across the split. Augmentation choices should reflect the instrument and biology: a transformation that is harmless for one modality may alter the meaning of another.

The original study demonstrates the combined architecture and training strategy in its biomedical settings. Its challenge results are evidence about those reported experiments, not a universal promise for every segmentation dataset. This page does not reproduce those experiments or report a modern benchmark. The executable checks validate narrower claims: the original spatial arithmetic, the crop operation and the numerical weighting equation.

What the original experiments establish

The electron-microscopy experiment used 30 annotated 512×512 training images from the Drosophila ventral nerve cord. Hidden test annotations were evaluated by the challenge organizers after thresholding membrane probability maps at ten levels. Averaging predictions over seven rotated inputs, the authors report warping error 0.0003529 and Rand error 0.0382. Table 1 rounds the first value to 0.000353. These are cited challenge results, not results rerun here. Original paper, §4 and Table 1

The ranking is metric-specific: the table is sorted by warping error, and another entry has a lower Rand error. Calling U-Net best on every reported measure would misread the evidence. Table 2 separately reports cell-segmentation IoU 0.9203 on PhC-U373 and 0.7756 on DIC-HeLa, with 35 and 20 partially annotated training images respectively. These tasks use different image modalities and a different evaluation measure from the EM challenge. Their numbers cannot be placed on one accuracy scale. Original paper, Table 2

The result supports the complete reported recipe in those datasets: architecture, annotation treatment, augmentation and evaluation procedure. It does not isolate an architecture-only causal effect, establish clinical validity, or promise the same sample efficiency under a new instrument. Prediction averaging is also part of the EM protocol; a single-view deployment should not silently inherit its reported score.

Original source, math and executable reference

The three rows below map original PDF evidence to a small standard-library reference. Each crop records the versioned PDF hash, page and rectangle. Code regions are synchronized from the downloadable source. The program deliberately contains no learned weights or convolution framework, making its checks easy to inspect without confusing a geometry demonstration with a trained model.

1. Derive the original spatial path

Original paper

Figure 1: original valid-convolution architecture

Figure 1 supplies the original sizes and channel widths.

Math and tensor shapes

\begin{aligned}n_{\rm convpair}&=n-4\\ n_{\rm pool}&=n/2\\ n_{\rm up}&=2n\end{aligned}

Starting at 572, exact pooling and valid convolution give a 28-pixel bottleneck and a 388-pixel output. The head has two original-paper classes. This executable reference tracks dimensions; it does not run learned convolutions.

Python implementation

unet.py · geometry
python
def geometry(size=572):
    """Track valid 3x3 pairs, exact 2x2 pooling and 2x2 up-convolutions."""
    if not isinstance(size, int) or isinstance(size, bool) or size < 188:
        raise ValueError("Input must be an integer at least 188")
    original, skips = size, []
    for level in range(4):
        size -= 4
        if size % 2:
            raise ValueError("Pooling must divide exactly at all four levels")
        skips.append({"size": size, "channels": 64 * 2**level})
        size //= 2
    size -= 4
    bottleneck, decoder = size, []
    for skip in reversed(skips):
        up = 2 * size
        crop = (skip["size"] - up) // 2
        size = up - 4
        decoder.append({"up": up, "crop": crop, "size": size,
                        "channels": skip["channels"],
                        "concatenated_channels": 2 * skip["channels"]})
    return {"input": original, "encoder": skips, "bottleneck": bottleneck,
            "decoder": decoder, "output": size, "halo": (original-size)//2}

2. Keep the aligned center of each skip

Original paper

Section 2: cropped skip concatenation

Section 2 specifies cropping the encoder map before concatenation.

Math and tensor shapes

\begin{aligned}A,B&\in\mathbb R^{H\times W\times C}\\ Y&=\operatorname{concat}_C(A,B)\\ Y&\in\mathbb R^{H\times W\times 2C}\end{aligned}

A and B denote the cropped skip and decoder maps; concat joins their C channel axes. A center crop removes borders independently in each channel; concatenation then joins feature channels, rather than adding their values. The helper uses floor for a documented odd margin. The original compatible geometry has symmetric integer crops.

Python implementation

unet.py · crop
python
def center_crop(values, height, width, target_height, target_width):
    """Crop a scalar channel; repeat independently for every skip channel."""
    dims = (height, width, target_height, target_width)
    if any(not isinstance(v, int) or isinstance(v, bool) or v <= 0 for v in dims):
        raise ValueError("Dimensions must be positive integers")
    if len(values) != height*width or target_height > height or target_width > width:
        raise ValueError("Invalid crop shape")
    top, left = (height-target_height)//2, (width-target_width)//2
    return [values[(top+y)*width+left+x]
            for y in range(target_height) for x in range(target_width)]

3. Calculate a boundary-focused pixel weight

Original paper

Equation 2: boundary weighting

Equation 2 defines the precomputed boundary weighting term.

Math and tensor shapes

\begin{aligned}s&=d_1+d_2\\ q&=s^2/(2\sigma^2)\\ w&=w_c+w_0e^{-q}\end{aligned}

The distances refer to the nearest and second-nearest distinct cell boundaries. Supplied distances isolate the equation; a real annotation pipeline must compute them. At large separation the extra term tends to zero, leaving the class-frequency weight.

Python implementation

unet.py · boundary
python
def boundary_weight(d1, d2, wc=1.0, w0=10.0, sigma=5.0):
    """Original boundary term plus a supplied class-frequency weight."""
    if not all(math.isfinite(v) for v in (d1, d2, wc, w0, sigma)):
        raise ValueError("Values must be finite")
    if min(d1, d2, wc, w0) < 0 or sigma <= 0:
        raise ValueError("Require nonnegative distances/weights and positive sigma")
    return wc + w0 * math.exp(-(d1+d2)**2/(2*sigma**2))

Original PDF excerpts: Olaf Ronneberger, Philipp Fischer, Thomas Brox. 1505.04597v1; cropped and rasterized without changing source colors. The manifest records source and image hashes.

Run python examples/unet.py --json from the repository, or download the reference. The self-check includes the canonical 572→388 path, compatible larger inputs, rectangular crop coordinates and boundary-weight limits. The README explains its scope; requirements contain no packages.

What to retain when using a modern U-Net

The enduring idea is a decoder that can combine coarse context with fine spatial evidence at multiple scales. To make that idea concrete in a new system, specify exactly where the tensors meet, how spatial sizes are matched, how channels are combined, what each output location represents, and how labels enter the loss. Those decisions are observable implementation choices.

A padded modern model may be easier to integrate into a pipeline because input and output sizes match. A multiclass projection can label several tissue types. An instance-aware objective can pursue a different output contract. None of these changes needs to be framed as an error; the error would be attributing them to the original architecture without saying what changed.

The combination of explicit geometry, local high-resolution evidence and deliberate training supervision remains a useful design vocabulary. It also provides a disciplined way to read later encoder-decoder papers: ask which information survives downsampling, which path restores access to detail, and which measurement supports the claimed improvement.

If you found this paper review helpful, consider sharing it with others.

Mastodon