What is RoI Pooling?
Region of Interest (RoI) pooling is a fundamental operation in two-stage object detectors like Faster R-CNN and Mask R-CNN. Given a CNN feature map and proposed regions of varying sizes, RoI pooling extracts fixed-size feature vectors for each region—enabling downstream classification and bounding box regression.
The challenge is that region proposals have arbitrary positions and sizes, but the detection head expects fixed-size inputs. RoI Pooling solved this with quantized max pooling, but its rounding errors became problematic for pixel-precise tasks. RoI Align eliminated quantization using bilinear interpolation, while Deformable RoI Pooling added learned offsets for shape-adaptive sampling.
The Problem: Arbitrary Regions, Fixed Networks
Helpful background
Convolution
Connect filters, channels, stride and padding to image feature maps and their output shapes.
Two-stage detectors face a fundamental mismatch:
- Region proposals come in arbitrary sizes (50×30, 200×150, 80×80...)
- Detection heads (FC layers) require fixed-size inputs (7×7×512)
We need to extract features from each proposed region and resize them to a fixed spatial size—but how do we handle regions that don't align with the feature map grid?
RoI Pooling: The Original Approach
Drag the teal rectangle, the actual RoI, across the feature map; with the rectangle focused, the arrow keys move it too. The dashed rose rectangle is the quantized RoI under this figure’s endpoint convention, the dashed amber bins show where each pooled value comes from, and the panel reports the signed change in area. This change alone does not measure alignment.
How RoI Pooling Works
-
Map RoI to Feature Map: Scale the RoI coordinates by the stride (e.g., 16x for VGG). A 160×96 RoI becomes 10×6 on the feature map.
-
Quantize Coordinates: Round floating-point coordinates to integers. This is the first source of quantization error.
-
Divide into Bins: Split the quantized region into a fixed grid (e.g., 7×7). Keep fractional bin sizes, floor each start and ceil each end, then clip to the feature map. Adjacent bins can overlap.
-
Max Pool Each Bin: Apply max pooling within each bin to produce the output feature.
The Two Levels of Quantization
RoI Pooling introduces two rounds of rounding:
- RoI Boundary Quantization: When mapping RoI coordinates to the feature map
- Bin Boundary Quantization: When selecting integer cells for each fractional bin
This figure floors both rectangle edges and treats the end as exclusive; its outlined cells match that half-open interval. This is a teaching convention: torchvision RoI Pool rounds endpoints and treats the end as inclusive before computing its bins. The resulting coordinates and pooled values can therefore differ between conventions. The per-bin floor/ceil and clipping shown here follow the same principle as that kernel.
RoI Align: Eliminating Quantization
Click or drag anywhere on the grid to move the sample point. The four nearest feature map cells are highlighted, and the panel shows each neighbor's weight and how the weights combine into the interpolated value.
RoI Align (Mask R-CNN, 2017) eliminates both quantization steps:
- No rounding of RoI coordinates
- Floating-point bin boundaries
- Regular sampling points within each bin (typically 4 per bin)
- Bilinear interpolation to compute values at exact positions
Bilinear Interpolation Formula
The following standalone interpolation example clamps all coordinates to the nearest border. It does not implement the complete RoI Align operator: torchvision’s aligned=True shifts scaled RoI coordinates by half a pixel, and its native kernel returns zero for samples sufficiently far outside the map.
import math
def bilinear_interpolate(feature_map, x, y):
"""
Compute feature value at floating-point position (x, y)
using the 4 nearest neighbors.
"""
height, width = feature_map.shape
x = min(max(x, 0.0), width - 1)
y = min(max(y, 0.0), height - 1)
x0, y0 = math.floor(x), math.floor(y)
x1, y1 = min(x0 + 1, width - 1), min(y0 + 1, height - 1)
dx, dy = x - x0, y - y0
# Distance weights
wa = (1 - dx) * (1 - dy) # weight for (x0, y0)
wb = dx * (1 - dy) # weight for (x1, y0)
wc = (1 - dx) * dy # weight for (x0, y1)
wd = dx * dy # weight for (x1, y1)
# Weighted sum
return (wa * feature_map[y0, x0] +
wb * feature_map[y0, x1] +
wc * feature_map[y1, x0] +
wd * feature_map[y1, x1])
Why This Matters for Segmentation
For bounding box detection, small misalignments might be acceptable. But for instance segmentation (predicting per-pixel masks), a 1-pixel shift can completely change which object a pixel belongs to. RoI Align's sub-pixel precision is essential for Mask R-CNN's performance.
Deformable RoI Pooling: Shape Adaptation
Choose an object shape, then switch between the regular sampling grid and the deformed one; the transition can be paused part-way. The offsets drawn here are predetermined illustrative formulas for each shape, not trained parameters. Each dot represents one illustrative sample per bin; the original operator applies the same offset to all samples within a bin.
Deformable RoI Pooling (Deformable ConvNets, 2017) augments RoI pooling with learned per-bin offsets. It retains the original discretized bin partition; it is not the RoI Align operator:
- An FC layer predicts a normalized 2D offset for each bin from pooled features.
- Each offset is scaled by the RoI width and height and shared by the samples in that bin.
- Bilinear interpolation evaluates shifted sample locations, and the values are averaged.
- Offsets are trained end-to-end via backpropagation.
This allows the network to adaptively focus on:
- Irregular object boundaries (non-rectangular shapes)
- Semantically important parts (face of a person, wheels of a car)
- Occluded regions (sampling around occlusions)
Method Comparison
Select a method to read its details, or select it again to clear the selection. The sections compare the same properties side by side, and the timeline shows the order in which the methods appeared.
| Aspect | RoI Pooling | RoI Align | Deformable RoI |
|---|---|---|---|
| Quantization | 2 levels | None | Discrete bins |
| Alignment | Quantized | Sub-pixel | Adaptive offsets |
| Shape Handling | Fixed grid | Fixed grid | Learned offsets |
| Speed | Fastest | Fast | Moderate |
| Offsets | None | None | 2 per bin |
| Best For | Classification | Segmentation | Complex shapes |
Use Cases
- Object Detection: Classification and localization of objects (Faster R-CNN with RoI Align)
- Instance Segmentation: Per-pixel masks for each object (Mask R-CNN requires RoI Align)
- Human Pose Estimation: Detecting keypoints on articulated bodies (Deformable RoI)
- Video Object Detection: Tracking objects across frames
- Medical Imaging: Detecting lesions with precise boundaries
- Autonomous Driving: Real-time detection of vehicles and pedestrians
Key Formulas
RoI Pooling (max pooling with quantization):
output[i,j] = max(features[⌊y1 + i×h/k⌋ : ⌈y1 + (i+1)×h/k⌉,
⌊x1 + j×w/k⌋ : ⌈x1 + (j+1)×w/k⌉])
RoI Align (bilinear sampling):
output[i,j] = avg(bilinear(features, sample_points[i,j]))
Deformable RoI (offset-augmented):
normalized_offsets = offset_predictor(pooled_features)
Δ[i,j] = γ × (w × normalized_offsets[i,j].x, h × normalized_offsets[i,j].y)
output[i,j] = avg(bilinear(features, (x1,y1) + p + Δ[i,j]) for p in discrete_bin[i,j])
Here γ is a fixed offset scale (0.1 in the paper); (x1,y1) is the RoI origin and p denotes a discrete position relative to that origin.
Best Practices
-
Use RoI Align by Default: Unless speed is critical, RoI Align's precision benefits outweigh its small overhead
-
Match Feature Map Resolution: Ensure RoI coordinates are correctly scaled by the backbone's stride (16 or 32)
-
Use Multiple Sampling Points: 4 samples per bin (2×2) is standard; more helps for small objects
-
Consider FPN for Multi-Scale: Assign RoIs to appropriate FPN levels based on size
-
Regularize Deformable Offsets: Add L2 regularization to prevent offsets from diverging
Further Reading
- Fast R-CNN - Original RoI Pooling
- Mask R-CNN - RoI Align for instance segmentation
- Deformable Convolutional Networks - Deformable RoI Pooling
- Feature Pyramid Networks - Multi-scale RoI assignment
- Cascade R-CNN - Multi-stage RoI refinement
Related concepts
Compare anchor-based vs anchor-free object detection: Faster R-CNN and RetinaNet anchors vs FCOS and CenterNet point-based methods.
Learn how DETR assigns unordered object targets to prediction slots, why greedy choices can fail, and how matching differs from training loss and evaluation.
Learn how normalized box coordinates, L1 loss, IoU, and Generalized IoU guide object localization, including disjoint boxes and containment limits.
Convert between xyxy, xywh and center-size boxes, normalize coordinates correctly, and avoid mistakes when resizing, clipping or flipping detection labels.
Follow a query from a spatial reference through learned offsets, bilinear feature sampling and weighted aggregation, with exact non-square coordinate arithmetic.
Learn how Feature Pyramid Networks build multi-scale feature representations through top-down pathways and lateral connections for robust object detection.
