Skip to main content

NAS-FPN: Learning to Design Feature Pyramid Networks

Understanding how neural architecture search discovers optimal feature pyramid architectures that outperform hand-designed alternatives

Overview

NAS-FPN asks a provocative question: what if we let an algorithm design the feature pyramid network instead of relying on human intuition? Using reinforcement learning to search over a vast space of possible architectures, NAS-FPN discovers irregular, asymmetric connection patterns that consistently outperform hand-designed alternatives like FPN and PANet—proving that the "obvious" top-down pathway wasn't optimal after all.

The key insight is that human designers favor symmetric, adjacent-scale connections because they're intuitive, but the optimal architecture often includes surprising long-range skip connections and asymmetric patterns that humans would never consider.

Key Concepts

Neural Architecture Search (NAS)

Automated discovery of optimal neural network architectures using search algorithms, typically reinforcement learning or evolutionary methods.

Merging Cell

The building block of NAS-FPN: takes two feature maps, resizes them to a target resolution, and combines them with sum or global-pooling attention.

Search Space

The set of all possible architectures that NAS can explore. Both inputs are selected without replacement from the growing set of features and earlier cell outputs, subject to resolution constraints.

Proxy Task

A simplified training task (subset of data, fewer epochs) used to quickly evaluate architectures during search. Rankings correlate with full training.

Proximal Policy Optimization (PPO)

The RL algorithm used to update the controller. High-AP architectures reinforce the decisions that generated them.

Stacking / Scalability

The discovered architecture can be repeated multiple times to trade compute for accuracy, enabling flexible deployment.

The Problem: Human Design Bias

Helpful background

Feature Pyramid Networks

Trace top-down and lateral connections, and relate feature-map resolution to object scale and RoI level assignment.

Open full page

Since Lin et al. introduced Feature Pyramid Networks in 2017, researchers have proposed numerous variants: PANet added bottom-up paths, BiFPN introduced weighted fusion. But all these designs share a limitation: they're constrained by human intuition about what connections "should" exist. Pick FPN, PANet or BiFPN to see the connections each draws between pyramid levels, then add NAS-FPN to compare it with the discovered pattern.

Human designers favor:

  • Symmetric patterns — if there's a top-down path, add a bottom-up path
  • Adjacent-scale connections — only connect P3↔P4, P4↔P5, etc.
  • Regular structures — same pattern repeated at each level

With five initial feature levels and multiple operations, the controller explores a large, constrained space of architectures.

The NAS-FPN Approach

NAS-FPN formulates feature pyramid design as a reinforcement learning problem. An RNN controller generates architecture specifications, child networks are trained and evaluated, and the controller is updated to favor high-performing designs. Press Play to run the loop from the current phase to the last, where it stops; pressing Play at the last phase starts again from the first with the next iteration, and Next moves one phase at a time.

How It Works

1

RNN Controller Generates Architecture

An LSTM-based controller outputs a sequence of decisions that specify the complete FPN architecture: which inputs to use, what operation to apply, where to output.

2

Build Child Network

The sampled architecture specification is instantiated as an actual neural network with the specified connections and operations.

3

Train on Proxy Task

Each child network is trained for 10 epochs on the proxy setup; AP on a held-out subset of COCO train2017 is the reward.

4

Compute Reward (AP)

The trained network's Average Precision on validation becomes the reward signal. Higher AP means better architecture.

5

Update Controller via PPO

PPO updates the controller policy using the observed AP reward, with a clipped objective limiting policy changes.

6

Search and select

The winning architecture first appeared at about 8,000 controller steps and was sampled again; the paper does not specify aggregate TPU-hours.

The Merging Cell: Building Block

The key abstraction in NAS-FPN is the merging cell: a unit that takes two input feature maps and combines them to produce one output. The controller decides the inputs, output resolution, and operation for each cell. Choose the two inputs, the output level and the operation to see how each input is resized to the output resolution before the merge.

For each merging cell, the controller makes four decisions:

  1. Input 1: Which feature level (P3, P4, P5, P6, or P7)
  2. Input 2: Which feature level or previous cell output
  3. Output resolution: What scale the merged output should be
  4. Binary operation: Sum (element-wise addition) or GP (global-pooling attention)

This simple abstraction enables a vast search space while keeping individual cells easy to implement and efficient to run.

The Discovered Architecture

After searching, NAS-FPN discovers architectures that look nothing like human designs. The connections are irregular, asymmetric, and include surprising long-range skip connections. Press Play to add merging cells one at a time from the current cell to the last (from the last it starts again from the blank start), or jump to a cell with the numbered buttons; Cell details lists the inputs, operation and output of each cell.

Key observations from the discovered architecture:

  • Non-adjacent connections: A refined P3 feature feeds the P5 output cell, skipping the P4 resolution
  • Asymmetric patterns: Different scales get different treatment
  • Strategic GP operations: Global-pooling attention used selectively, not uniformly (cells 1, 5, 6 and 7)
  • Long-range dependencies: Features can skip intermediate resolutions

These patterns emerge because they improve detection accuracy, not because they're intuitive to humans.

Scalability: Stacking for Performance

One elegant property of NAS-FPN: the discovered architecture can be stacked multiple times to trade compute for accuracy. Each repetition refines the feature pyramid further. Choose how many times the architecture is stacked to see the repeated structure.

Real-World Applications

Maximum Detection Accuracy

When accuracy matters more than inference speed

Use NAS-FPN ×7 with strong backbone for SOTA results

Scalable Deployment

Need different accuracy-speed trade-offs for different devices

Deploy ×1 on edge, ×5 on server

Drop-in FPN Replacement

Existing detector with standard FPN neck

Replace FPN with NAS-FPN, keep everything else

Research Baseline

Studying what makes FPN architectures effective

Analyze discovered patterns to inform future designs

AutoML Pipelines

Building automated detection systems

Use NAS-FPN as the neck in NAS-based detectors

Transfer to New Domains

Applying to medical imaging, satellite, etc.

Architecture generalizes; may need domain-specific search

Advantages & Limitations

Advantages

  • ✓Discovers architectures humans wouldn't consider
  • ✓+2.9 AP over standard FPN in the matched ResNet-50 comparison
  • ✓Scalable via stacking for accuracy-compute trade-offs
  • ✓Drop-in replacement for existing FPN implementations
  • ✓Reveals insights about optimal feature pyramid design
  • ✓Architecture generalizes across different backbones

Limitations

  • ×Search uses a 100-TPU worker pool; aggregate device-hours unspecified
  • ×Discovered architecture is less interpretable than hand-designed
  • ×May not be optimal for domains very different from COCO
  • ×More parameters and compute than simple FPN
  • ×Search space design requires expertise
  • ×Results depend on proxy task correlation

Best Practices

  • Start with Published Architecture: Use the paper's discovered architecture directly—no need to re-run search. The 7-cell design works well across domains.
  • Choose Stacking Based on Budget: ×1 for real-time, ×3-5 for balanced, ×7 for maximum accuracy. Profile on your target hardware.
  • Keep Strong Backbone: NAS-FPN amplifies backbone quality. ResNet-50 gives 39.9 AP; AmoebaNet pushes to 48.3 AP.
  • Use with RetinaNet: The paper validates on RetinaNet. Results should transfer to other one-stage detectors.
  • Consider BiFPN for Efficiency: If you need maximum efficiency over accuracy, EfficientDet's BiFPN may be better suited.
  • Analyze for Insights: Study the discovered connections to understand what patterns matter—may inspire better hand-designed alternatives.

NAS-FPN vs Other FPN Variants

AspectFPNPANetBiFPNNAS-FPN
Design MethodHand-craftedHand-craftedHand-craftedNAS (RL)
Connection PatternTop-downTop-down + Bottom-upBidirectional + SkipDiscovered (irregular)
Fusion MethodAddAddWeighted AddSum or GP (learned)
ScalabilityFixedFixedStackableStackable (×1-7)
Search CostN/AN/AN/ANot specified

Lessons from NAS-FPN

Beyond the specific architecture, NAS-FPN teaches broader lessons:

Human intuitions are biased: We favor symmetric, "clean" architectures even when irregular ones work better. The search reveals that our design preferences don't align with optimal performance.

Skip connections matter more than adjacency: Direct P3→P5 connections can be more valuable than going through P4. Information doesn't need to flow through every intermediate level.

Search space design is crucial: The merging cell abstraction enables tractable search while being expressive enough to discover novel architectures.

Proxy tasks work: Quick training on small data correlates with full training, enabling fast architecture evaluation. This insight enables efficient search.

Scalability should be designed in: The best architectures can be stacked for accuracy-compute trade-offs, providing deployment flexibility.

Performance Summary

The paper's Table 1 reports RetinaNet on COCO test-dev, using ResNet-50, 256 feature channels and 640 × 640 input:

PyramidAPTotal FLOPsTotal parameters
FPN37.0193.6B34.0M
NAS-FPN ×739.9281.3B60.3M

The matched improvement is +2.9 AP. These FLOPs and parameters describe the entire detector, not just its neck. Source: NAS-FPN paper, Table 1.

Further Reading

If you found this explanation helpful, consider sharing it with others.

Mastodon