Skip to main content

Feature Pyramid Networks: Semantics at Every Scale

Build FPN from C2–C5, lateral 1×1 projections, nearest top-down addition and 3×3 outputs; trace RoI level assignment and the original detection evidence.

TL;DR

  • FPN builds prediction features from one backbone pass, combining coarse semantic context with finer spatial evidence through top-down upsampling and lateral connections.
  • Lateral 1×1 projections make channel widths compatible for addition. The merged inner maps feed the top-down recurrence; separate 3×3 convolutions produce the final pyramid outputs.
  • FPN supplies features inside a detector. RPN anchor scales and area-based RoI level assignment serve different stages, and selecting a map is separate from pooling or aligning a region on it.
  • The cited COCO ablations support the full construction under specified detector protocols. The scalar reference checks tensor and assignment contracts rather than learned semantics, detection AP or deployment speed.

Why combine feature scales

A small object can occupy only a few cells in a detector's deepest feature map. Earlier maps retain more spatial detail, but their features have passed through fewer layers of semantic processing. Feature Pyramid Networks connect those representations: top-down features bring context to high-resolution maps, and lateral features supply the local evidence at the matching scale.

This explanation follows FPN v2, the April 2017 revision of the CVPR 2017 paper. All interactive arrays and dimensions below are authored analytical examples. They are neither recorded features from a trained ResNet nor local detector benchmarks. The original PDF excerpts are source evidence, and the cited experimental numbers retain their evaluation protocols.

A backbone hierarchy already has multiple scales

A convolutional backbone progressively reduces spatial resolution. In a standard ResNet naming scheme, C2, C3, C4 and C5 refer to outputs from successive stages, with nominal strides four, eight, sixteen and thirty-two relative to the image. They contain different spatial grids and different channel widths.

It is tempting to call this hierarchy a feature pyramid and attach a prediction head to every stage. But resolution alone does not make the representations equally useful. Earlier features are relatively local and shallow; later ones have undergone more transformations and aggregate wider context. A small object needs a sufficiently fine grid, while recognition also benefits from semantic information.

Another route is an image pyramid: resize the input image several times and run a backbone at each size. That supplies semantically deep features at several effective scales, but it repeats much of the expensive image computation. FPN instead builds additional maps from one backbone pass. Original paper, Figure 1 and §3

One backbone; several prediction scales

Calculated nominal ResNet stage shapes for a single input image. C2–C5 are backbone maps, I2–I5 are the inner maps, and P2–P5 are separate 256-channel outputs.

800
1280

Start at I5 = 1×1(C5). At each finer level, add the resized higher inner map to the lateral projection. Each inner map also branches right into its own 3×3 output filter.

C5

25 × 40

stride 32

1×1 project

I5

base from C5

no addition

256 ch

3×3 output

P5

25 × 40

256 ch

Upsample I5

to C4 size

C4

50 × 80

stride 16

1×1 project

I4

+

lateral + top-down

256 ch

3×3 output

P4

50 × 80

256 ch

Upsample I4

to C3 size

C3

100 × 160

stride 8

1×1 project

I3

+

lateral + top-down

256 ch

3×3 output

P3

100 × 160

256 ch

Upsample I3

to C2 size

C2

200 × 320

stride 4

1×1 project

I2

+

lateral + top-down

256 ch

3×3 output

P2

200 × 320

256 ch

P2 retains 64000 spatial positions; P5 has 1000. All four have equal channel width, allowing shared detection heads.

The listed shapes use ceil(image/stride). Exact backbone padding remains an implementation choice. The original RPN also uses P6 by stride-two subsampling P5; the Fast R-CNN region head uses P2–P5.

The distinction between C and P is essential. C2–C5 are backbone outputs; P2–P5 are the constructed prediction features. An FPN does not rename the original tensors and declare the semantic gap solved. It transforms and combines them. In the original design, all P maps have 256 channels, even though their heights and widths differ.

Equal channel width supports a shared head across pyramid levels. It does not make the feature values or receptive fields identical. A location on P2 and one on P5 correspond to different sampling densities in the original image, so scale-specific placement remains part of detection.

Make the lateral and top-down features compatible

At the coarsest level, project C5 with a 1×1 convolution. To move down one level, upsample the coarser inner representation using nearest-neighbor sampling, project the corresponding backbone map with another 1×1 convolution, and add the aligned tensors elementwise. Repeat until reaching the finest desired level. Original paper, Figure 3 and §3

The 1×1 projection changes channels at each spatial location. It can learn combinations of channels without mixing neighboring positions. Its immediate structural role is to put the lateral map into the same channel space and width as the top-down representation. The learned weights determine which evidence that representation carries.

Nearest upsampling changes the spatial grid by repeating coarse values. It cannot independently invent high-frequency information about the image. The lateral branch is valuable precisely because it contributes information computed on the finer backbone grid. The subsequent learned operations can combine that local evidence with the propagated context.

Using T to name the internal merged map makes the computation unambiguous:

T_5=\operatorname{Conv}_{1\times1}(C_5),\qquad T_l=\operatorname{Conv}_{1\times1}(C_l)+\operatorname{Up}(T_{l+1})

This is addition, rather than the skip concatenation used in the original U-Net. Addition retains 256 channels. Concatenation would increase channels and require a changed downstream operation. The two designs address related multiscale questions with different tensor contracts.

Produce an output after each merge

The original FPN adds a 3×3 convolution to each merged map to form the final P output. The paper motivates it as reducing aliasing effects from upsampling. A learned 3×3 filter can mix neighboring values and channels; it is not equivalent to simply relabeling the inner map. Original paper, §3

P_l=\operatorname{Conv}_{3\times3}(T_l)

Add aligned features, then produce a 3×3 output

Authored scalar C2–C5 maps have constant values 1, 2, 3 and 4. A scalar multiplier stands in for learned lateral projections; a fixed mean stands in for learned 3×3 output filters.

1

Inner I2: lateral + resized I3

10.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.010.0

Output P2: illustrative zero-padded 3×3 mean

4.46.76.76.76.76.74.46.710.010.010.010.010.06.76.710.010.010.010.010.06.76.710.010.010.010.010.06.74.46.76.76.76.76.74.4

At the first location: 1.00 lateral + 9.00 top-down = 10.00 before the output filter.

Top-down propagation uses the merged inner map, before output smoothing. This is not a trained semantic representation. Zero padding makes the illustrative 3×3 mean smaller at borders.

The scalar explorer deliberately makes these steps inspectable. The supplied C2–C5 maps contain constant values one, two, three and four. A scalar multiplier replaces a learned lateral projection. Top-down addition therefore yields an inner value of ten at C2's resolution when all paths are present. The displayed 3×3 mean is an authored replacement for the learned output convolution, and zero padding reduces its boundary values.

This experiment exposes an easy implementation mistake: the top-down recurrence uses T, the merged internal map, rather than repeatedly propagating the already filtered P output. Passing the mean-filtered scalar output downward would spread boundary attenuation into subsequent levels. A real learned network would also change. Writing the two operations separately makes their roles easier to test.

Odd image dimensions introduce another practical issue. A coarse feature map doubled exactly may be one cell larger than the lateral feature map, depending on backbone rounding. The reference resizes to the explicit lateral target shape using a stated floor-index nearest mapping. A production implementation should make its convention equally clear. Shape matching by an undocumented crop or interpolation option can change sampling behavior along the right and bottom edges.

FPN is a feature extractor inside a detector

FPN does not replace the full Faster R-CNN system. A region proposal network still evaluates candidate objects, classification and box regression still require heads, and a region-based detector still extracts features for proposals. The pyramid changes which features those components consume and where they operate. Original paper, §4

For the RPN application, the paper distributes anchor areas across pyramid levels. Its P2–P6 levels cover nominal square anchor areas from 32² through 512² pixels. An anchor is a reference box; its scale is not the same thing as a feature map's width or a guaranteed object size. Anchor matching and box regression connect these references to annotated objects during training.

The original P6 deserves a precise qualification. It is introduced as stride-two subsampling of P5 to cover the largest RPN anchor scale. The Fast R-CNN region head in the next application does not use P6. Later detectors may construct additional levels with different convolutions; those choices should not be retroactively attributed to this original RPN configuration.

Similarly, the region-level selection rule is different from choosing which anchors are positive. One rule selects a map for a proposed region. The other assigns supervision to dense reference boxes. They both involve scale, but they belong to different stages of the detection pipeline.

Assign an RoI by its square-root area

A proposal can be pooled from one of P2–P5. The paper chooses its level using image-space width w and height h:

k=\left\lfloor k_0+\log_2\left(\frac{\sqrt{wh}}{224}\right)\right\rfloor,\qquad k_0=4

The result is clipped to the available levels. A 224×224 region selects P4; doubling both axes moves to P5. Halving both axes moves to P3. The square-root area supplies a scalar object scale while permitting rectangular regions. Original paper, Equation 1

Choose the RoI feature level by area

The original Fast R-CNN application maps a region's square-root area to one of P2–P5. This chooses which feature map to pool from, not which category the region contains.

224
224

P2

stride 4

P3

stride 8

P4

stride 16

P5

stride 32

√(w h) = 224.00 → floor(4 + log₂(scale/224)) = 4 → P4

Equal areas receive equal levels even with different aspect ratios. A 112×448 RoI and a 224×224 RoI both select P4.

Calculated original-paper assignment with k₀=4 and canonical scale 224 pixels. The floor creates intentional scale boundaries; clipping covers objects outside the level range. This differs from RPN anchor assignment.

Aspect ratio does not enter independently. A 112×448 proposal has the same area as a 224×224 proposal and therefore receives the same level. This is a useful test case: choosing a level from the longest side would implement a different rule. The floor also creates an intentional discontinuity at scale boundaries, which should not be confused with quantizing RoI sample coordinates.

After choosing a level, RoI pooling produces the fixed-size representation expected by the head. FPN answers which map supplies features; a pooling or alignment method answers where and how the region is sampled on that map. This separation is why FPN and RoIAlign can be used together without being competing solutions to the same problem.

For very small or large regions, clipping prevents indexing outside the available pyramid. It does not guarantee the chosen resolution is ideal for every object. Small-object recognition still depends on image information, backbone sampling, annotation quality and the head. The scale heuristic is a concrete engineering rule, not a proof of scale invariance.

Read the evidence with the detector held in view

Table 2 provides a useful controlled experiment: ResNet-50 models train on COCO trainval35k, evaluate on minival, and share a fixed set of FPN-RPN proposals. The C4 baseline obtains 31.9 box AP; the full pyramid obtains 33.9 AP, with small-object AP increasing from 15.7 to 17.8. Removing top-down connections gives 24.9 AP; removing lateral connections gives 31.3 AP in those specified variants. Original paper, Table 2

Holding proposals fixed helps inspect the region-feature contribution. It also means Table 2 does not measure the complete change from a single-level RPN to a pyramid RPN. Table 3 evaluates Faster R-CNN with the proposal backbone made consistent with the detector. Mixing these two tables without stating that distinction obscures what has been controlled.

The ablations support the value of the complete feature construction in the reported setting. They do not make “more scales” an unconditional explanation: the lateral-only bottom-up variant performs poorly, despite retaining multiple resolutions. Conversely, the full FPN's single finest P2 output reaches 33.4 AP in Table 2, close to the complete 33.9 AP. That observation makes the contribution richer than a slogan about four prediction grids. The enriched high-resolution features themselves matter.

These values are cited box-detection AP under COCO's protocol. They are not mask AP, pixel accuracy or a result of running the scalar program. The paper's original speed claims are hardware- and implementation-specific; this review does not extrapolate them into a contemporary latency promise. Memory traffic, high-resolution activation storage and detector heads remain substantial costs even when the backbone is shared across scales.

Original source beside executable arithmetic

The reference isolates three contracts: explicit-size nearest sampling, inner-map lateral addition followed by an output operator, and area-based RoI level assignment. It neither learns semantic features nor runs a detector. Each row below includes the relevant original PDF excerpt with versioned provenance.

1. Resize the higher map to the lateral shape

Original paper

Figure 3: top-down upsampling and lateral addition

Figure 3 connects the top-down upsampling path to a lateral projection.

Math and tensor shapes

\begin{aligned}i&=\lfloor yH_s/H_t\rfloor\\ j&=\lfloor xW_s/W_t\rfloor\\ U[y,x]&=T_{l+1}[i,j]\end{aligned}

Nearest sampling repeats coarse feature values. Explicit target dimensions avoid off-by-one failures on odd rectangles. This reference uses floor-index nearest mapping; it does not claim that every resize API has identical coordinate rules.

Python implementation

fpn.py · upsample
python
def nearest_resize(values, height, width, target_height, target_width):
    """Use floor-index nearest sampling with explicit target shape for odd grids."""
    dims = (height, width, target_height, target_width)
    if any(not isinstance(v, int) or isinstance(v, bool) or v <= 0 for v in dims):
        raise ValueError("Dimensions must be positive integers")
    if len(values) != height*width or not all(math.isfinite(v) for v in values):
        raise ValueError("Feature shape/value mismatch")
    return [values[min(height-1, y*height//target_height)*width
                   + min(width-1, x*width//target_width)]
            for y in range(target_height) for x in range(target_width)]

2. Merge inner maps, then filter each output

Original paper

Section 3: 1×1 lateral projection and 3×3 outputs

Section 3 specifies lateral 1×1 projections, elementwise addition and final 3×3 filters.

Math and tensor shapes

\begin{aligned}A_l&=\operatorname{Conv}_{1\times1}(C_l)\\ T_5&=A_5\\ T_l&=A_l+\operatorname{Up}(T_{l+1})\\ &\quad(l=2,3,4)\\ P_l&=\operatorname{Conv}_{3\times3}(T_l)\\ &\quad(l=2,3,4,5)\end{aligned}

All real lateral maps have 256 channels. The readable scalar surrogate substitutes a scale and a fixed mean for learned convolutions. It propagates the merged inner map top-down; feeding the smoothed output back changes the computation.

Python implementation

fpn.py · fusion
python
def build_pyramid(levels, lateral=1.0, top_down=True):
    """Propagate inner maps; apply the 3x3 output operator after each merge."""
    if len(levels) != 4 or not math.isfinite(lateral):
        raise ValueError("Supply C2, C3, C4, C5 and finite lateral scale")
    output, higher = [None]*4, None
    for index in range(3, -1, -1):
        height, width, values = levels[index]
        projected = [lateral*v for v in nearest_resize(values, height, width,
                                                       height, width)]
        up = (nearest_resize(higher["inner"], higher["height"], higher["width"],
                             height, width) if higher and top_down
              else [0.0]*len(values))
        inner = [a+b for a, b in zip(projected, up)]
        higher = {"level": index+2, "height": height, "width": width,
                  "inner": inner, "values": smooth(inner, height, width)}
        output[index] = higher
    return output

3. Select the Fast R-CNN region level

Original paper

Equation 1: RoI-to-pyramid assignment and P6 scope

Equation 1 assigns an RoI to a level; the footnote restricts P6 to the RPN.

Math and tensor shapes

\begin{aligned}s&=\frac{\sqrt{wh}}{224}\\ k_0&=\lfloor4+\log_2s\rfloor\\ k&=\operatorname{clip}_{[2,5]}(k_0)\end{aligned}

Use width and height in image pixels. The square-root area chooses a spatial feature scale, while subsequent RoI pooling fixes the output tensor dimensions. The original P6 is stride-two subsampling of P5 for the 512² RPN anchor scale, not an RoI level.

Python implementation

fpn.py · assignment
python
def roi_level(width, height):
    """Assign Fast R-CNN RoIs to P2..P5; P6 belongs to the original RPN."""
    if not all(math.isfinite(v) and v > 0 for v in (width, height)):
        raise ValueError("Positive finite RoI extent required")
    scale = math.sqrt(width*height)
    raw = math.floor(4+math.log2(scale/224))
    return {"raw": raw, "level": min(5, max(2, raw)), "scale": scale}

Original PDF excerpts: Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie. 1612.03144v2; crop geometry and source/image hashes are recorded in the manifest.

Run python examples/fpn.py --json, or download the reference. The README and requirements explain the standard-library-only environment. Checks include odd rectangular grids, singleton maps, constant-field propagation, boundary padding and exact scale transitions.

When adapting an FPN, retain this habit of naming each tensor's role. Specify the backbone stages, lateral channel count, upsampling coordinates, merge operation, output filters, extra levels and head assignments. The architecture then becomes a testable graph of operations instead of a diagram whose arrows hide incompatible assumptions.

If you found this paper review helpful, consider sharing it with others.

Mastodon