Skip to main content

RoI Pooling, RoI Align & Deformable RoI Pooling

Understanding region-based feature extraction for object detection, from quantized pooling to sub-pixel alignment and adaptive sampling

What is RoI Pooling?

Region of Interest (RoI) pooling is a fundamental operation in two-stage object detectors like Faster R-CNN and Mask R-CNN. Given a CNN feature map and proposed regions of varying sizes, RoI pooling extracts fixed-size feature vectors for each region—enabling downstream classification and bounding box regression.

The challenge is that region proposals have arbitrary positions and sizes, but the detection head expects fixed-size inputs. RoI Pooling solved this with quantized max pooling, but its rounding errors became problematic for pixel-precise tasks. RoI Align eliminated quantization using bilinear interpolation, while Deformable RoI Pooling added learned offsets for shape-adaptive sampling.

The Problem: Arbitrary Regions, Fixed Networks

Helpful background

Convolution

Connect filters, channels, stride and padding to image feature maps and their output shapes.

Open full page

Two-stage detectors face a fundamental mismatch:

  1. Region proposals come in arbitrary sizes (50×30, 200×150, 80×80...)
  2. Detection heads (FC layers) require fixed-size inputs (7×7×512)

We need to extract features from each proposed region and resize them to a fixed spatial size—but how do we handle regions that don't align with the feature map grid?

RoI Pooling: The Original Approach

Drag the teal rectangle, the actual RoI, across the feature map; with the rectangle focused, the arrow keys move it too. The dashed rose rectangle is the quantized RoI under this figure’s endpoint convention, the dashed amber bins show where each pooled value comes from, and the panel reports the signed change in area. This change alone does not measure alignment.

How RoI Pooling Works

  1. Map RoI to Feature Map: Scale the RoI coordinates by the stride (e.g., 16x for VGG). A 160×96 RoI becomes 10×6 on the feature map.

  2. Quantize Coordinates: Round floating-point coordinates to integers. This is the first source of quantization error.

  3. Divide into Bins: Split the quantized region into a fixed grid (e.g., 7×7). Keep fractional bin sizes, floor each start and ceil each end, then clip to the feature map. Adjacent bins can overlap.

  4. Max Pool Each Bin: Apply max pooling within each bin to produce the output feature.

The Two Levels of Quantization

RoI Pooling introduces two rounds of rounding:

  1. RoI Boundary Quantization: When mapping RoI coordinates to the feature map
  2. Bin Boundary Quantization: When selecting integer cells for each fractional bin

This figure floors both rectangle edges and treats the end as exclusive; its outlined cells match that half-open interval. This is a teaching convention: torchvision RoI Pool rounds endpoints and treats the end as inclusive before computing its bins. The resulting coordinates and pooled values can therefore differ between conventions. The per-bin floor/ceil and clipping shown here follow the same principle as that kernel.

RoI Align: Eliminating Quantization

Click or drag anywhere on the grid to move the sample point. The four nearest feature map cells are highlighted, and the panel shows each neighbor's weight and how the weights combine into the interpolated value.

RoI Align (Mask R-CNN, 2017) eliminates both quantization steps:

  1. No rounding of RoI coordinates
  2. Floating-point bin boundaries
  3. Regular sampling points within each bin (typically 4 per bin)
  4. Bilinear interpolation to compute values at exact positions

Bilinear Interpolation Formula

The following standalone interpolation example clamps all coordinates to the nearest border. It does not implement the complete RoI Align operator: torchvision’s aligned=True shifts scaled RoI coordinates by half a pixel, and its native kernel returns zero for samples sufficiently far outside the map.

python
import math

def bilinear_interpolate(feature_map, x, y):
    """
    Compute feature value at floating-point position (x, y)
    using the 4 nearest neighbors.
    """
    height, width = feature_map.shape
    x = min(max(x, 0.0), width - 1)
    y = min(max(y, 0.0), height - 1)
    x0, y0 = math.floor(x), math.floor(y)
    x1, y1 = min(x0 + 1, width - 1), min(y0 + 1, height - 1)
    dx, dy = x - x0, y - y0

    # Distance weights
    wa = (1 - dx) * (1 - dy)  # weight for (x0, y0)
    wb = dx * (1 - dy)  # weight for (x1, y0)
    wc = (1 - dx) * dy  # weight for (x0, y1)
    wd = dx * dy  # weight for (x1, y1)

    # Weighted sum
    return (wa * feature_map[y0, x0] +
            wb * feature_map[y0, x1] +
            wc * feature_map[y1, x0] +
            wd * feature_map[y1, x1])

Why This Matters for Segmentation

For bounding box detection, small misalignments might be acceptable. But for instance segmentation (predicting per-pixel masks), a 1-pixel shift can completely change which object a pixel belongs to. RoI Align's sub-pixel precision is essential for Mask R-CNN's performance.

Deformable RoI Pooling: Shape Adaptation

Choose an object shape, then switch between the regular sampling grid and the deformed one; the transition can be paused part-way. The offsets drawn here are predetermined illustrative formulas for each shape, not trained parameters. Each dot represents one illustrative sample per bin; the original operator applies the same offset to all samples within a bin.

Deformable RoI Pooling (Deformable ConvNets, 2017) augments RoI pooling with learned per-bin offsets. It retains the original discretized bin partition; it is not the RoI Align operator:

  1. An FC layer predicts a normalized 2D offset for each bin from pooled features.
  2. Each offset is scaled by the RoI width and height and shared by the samples in that bin.
  3. Bilinear interpolation evaluates shifted sample locations, and the values are averaged.
  4. Offsets are trained end-to-end via backpropagation.

This allows the network to adaptively focus on:

  • Irregular object boundaries (non-rectangular shapes)
  • Semantically important parts (face of a person, wheels of a car)
  • Occluded regions (sampling around occlusions)

Method Comparison

Select a method to read its details, or select it again to clear the selection. The sections compare the same properties side by side, and the timeline shows the order in which the methods appeared.

AspectRoI PoolingRoI AlignDeformable RoI
Quantization2 levelsNoneDiscrete bins
AlignmentQuantizedSub-pixelAdaptive offsets
Shape HandlingFixed gridFixed gridLearned offsets
SpeedFastestFastModerate
OffsetsNoneNone2 per bin
Best ForClassificationSegmentationComplex shapes

Use Cases

  • Object Detection: Classification and localization of objects (Faster R-CNN with RoI Align)
  • Instance Segmentation: Per-pixel masks for each object (Mask R-CNN requires RoI Align)
  • Human Pose Estimation: Detecting keypoints on articulated bodies (Deformable RoI)
  • Video Object Detection: Tracking objects across frames
  • Medical Imaging: Detecting lesions with precise boundaries
  • Autonomous Driving: Real-time detection of vehicles and pedestrians

Key Formulas

RoI Pooling (max pooling with quantization):

text
output[i,j] = max(features[⌊y1 + i×h/k⌋ : ⌈y1 + (i+1)×h/k⌉,
                          ⌊x1 + j×w/k⌋ : ⌈x1 + (j+1)×w/k⌉])

RoI Align (bilinear sampling):

text
output[i,j] = avg(bilinear(features, sample_points[i,j]))

Deformable RoI (offset-augmented):

text
normalized_offsets = offset_predictor(pooled_features)
Δ[i,j] = γ × (w × normalized_offsets[i,j].x, h × normalized_offsets[i,j].y)
output[i,j] = avg(bilinear(features, (x1,y1) + p + Δ[i,j]) for p in discrete_bin[i,j])

Here γ is a fixed offset scale (0.1 in the paper); (x1,y1) is the RoI origin and p denotes a discrete position relative to that origin.

Best Practices

  1. Use RoI Align by Default: Unless speed is critical, RoI Align's precision benefits outweigh its small overhead

  2. Match Feature Map Resolution: Ensure RoI coordinates are correctly scaled by the backbone's stride (16 or 32)

  3. Use Multiple Sampling Points: 4 samples per bin (2×2) is standard; more helps for small objects

  4. Consider FPN for Multi-Scale: Assign RoIs to appropriate FPN levels based on size

  5. Regularize Deformable Offsets: Add L2 regularization to prevent offsets from diverging

Further Reading

If you found this explanation helpful, consider sharing it with others.

Mastodon