Skip to main content

Feature Pyramid Networks (FPN) Explained

Learn how Feature Pyramid Networks build multi-scale feature representations through top-down pathways and lateral connections for robust object detection.

Overview

Feature Pyramid Networks (FPN) solved one of computer vision's oldest headaches: detecting objects at wildly different scales without paying the computational cost of image pyramids. Before FPN, you either ran a detector on multiple resized copies of the image (slow) or used only the final CNN features (poor for small objects). FPN showed that a lightweight top-down pathway could recycle features the backbone already computed, producing a rich multi-scale representation at marginal extra cost.

The key innovation is combining semantically strong but spatially coarse features from deep layers with spatially precise but semantically weak features from shallow layers — giving every pyramid level the best of both worlds.

The Map Zoom Analogy

Think of a mapping application. When you zoom all the way out, you see country borders and major highways — high-level structure, no detail. When you zoom in, you see individual buildings and street names — rich detail, no context. Now imagine a system that overlays the zoomed-out labels onto the zoomed-in view, giving you both detail and context simultaneously. That is exactly what FPN does with CNN features: it takes the "zoomed-out" semantic understanding from deep layers and fuses it back into the "zoomed-in" spatial detail of shallow layers.

Pick a zoom level to see how its grid, resolution, receptive field and detected objects change, and which FPN level it corresponds to. Switching zoom levels in a map reveals different features in the same way that FPN creates a pyramid of feature maps at multiple resolutions: each level is responsible for detecting objects at a specific scale, ensuring nothing is missed regardless of size.

FPN Architecture

Bottom-Up Pathway

The bottom-up pathway is simply the backbone network (ResNet, EfficientNet, etc.) running its normal forward pass, the same kind of deep CNN backbone introduced in Faster R-CNN for region-based detection. As features flow through successive stages, spatial resolution halves while channel depth and semantic richness increase. FPN taps into the output of each stage, producing feature maps C_2, C_3, C_4, C_5 at strides of 4, 8, 16, and 32 pixels respectively.

Top-Down Pathway

Starting from the coarsest level C_5, FPN first projects it to initialize the inner feature. It then upsamples that inner feature by 2x using nearest-neighbor interpolation and merges it with the next finer lateral projection:

M_5 = C_5^{\text{lateral}}, \qquad M_i = C_i^{\text{lateral}} + \text{Upsample}(M_{i+1}) \quad (i=2,3,4)
P_i = \text{Conv}_{3 \times 3}(M_i)

Here C_i^{\text{lateral}} is the backbone feature passed through a 1x1 convolution. The inner feature M_i, rather than the smoothed output P_i, is propagated to the next level. The separate 3x3 output convolution reduces aliasing artifacts from upsampling.

ROI Level Assignment

Helpful background

RoI Pooling and RoI Align

Turn arbitrary regions into fixed-size features and inspect how rounding, interpolation and sampling offsets affect alignment.

Open full page

Objects of different sizes are assigned to the pyramid level best suited for their scale:

k = \lfloor k_0 + \log_2\!\bigl(\sqrt{wh}\,/\,224\bigr) \rfloor

Where k_0 = 4 is the canonical level, and w, h are the ROI dimensions. The resulting level is clamped to the available range P2–P5. Small objects land on fine-grained levels; large objects land on coarse levels.

Exploring the Architecture

The interactive diagram below lets you trace how features flow through FPN's three pathways — bottom-up, lateral, and top-down. Select any pyramid level to see its spatial resolution, channel depth, and which backbone stage feeds into it. Press Play to animate the flow: inner features M are propagated, and each produces a corresponding P output. Choose a level with the buttons or by clicking it in the diagram.

How Lateral Connections Work

Lateral connections are the bridge between the bottom-up and top-down pathways. A 1x1 convolution projects each backbone stage's feature map to a uniform channel dimension (typically 256), then element-wise addition merges it with the upsampled top-down feature. This fusion is what gives each pyramid level both spatial precision and semantic richness — the shallow feature contributes where objects are, while the deep feature contributes what objects are. Pick a level, then press Play to animate the merge; each stage is described under the diagram.

Multi-Scale Object Detection

FPN provides feature maps at several resolutions for objects of different sizes. The following authored illustration applies the ROI assignment rule and shows hypothetical confidence scores; it does not run a detector or measure recall. Switch between single-scale detection at P4 and FPN multi-scale to compare how each illustrated object is assigned, and use the object size slider to apply the level-assignment formula to one size.

FPN Variants

Since the original FPN, researchers have proposed many variations that improve feature fusion quality, speed, or both. PANet adds a second bottom-up pass for better localization. BiFPN introduces learnable fusion weights and skip connections. NAS-FPN uses neural architecture search to discover optimal connection patterns. The rows below compare these variants across accuracy, speed, and architectural complexity.

Common Pitfalls

1. Feature Misalignment After Upsampling

Nearest-neighbor upsampling can introduce spatial misalignment between the top-down and lateral features, especially at boundaries. This manifests as blurry detections or shifted bounding boxes. Deformable convolutions in the lateral path or aligned bilinear upsampling can mitigate this.

2. Channel Dimension Mismatch

All pyramid levels must share the same channel dimension for downstream heads to work. If the backbone stages have very different channel counts, the 1x1 lateral convolutions must be initialized carefully — poor initialization can cause some levels to dominate while others contribute noise.

3. Ignoring Extra Pyramid Levels

The original FPN stops at P5, but many detection frameworks add P6 and P7 via strided convolutions for detecting very large objects. Omitting these extra levels when your dataset contains large objects (vehicles in aerial imagery, buildings in satellite photos) leaves performance on the table.

Further Reading

If you found this explanation helpful, consider sharing it with others.

Mastodon