TL;DR
- Mask R-CNN adds a spatial mask branch beside classification and box refinement, predicting a separate mask for each detected instance.
- RoIAlign preserves fractional region and sample coordinates, then interpolates features. Correct alignment depends on the coordinate convention, not interpolation alone.
- A positive region supervises only its ground-truth class’s binary mask channel; inference selects the mask using the predicted class.
- The paper’s controlled COCO ablation supports the alignment mechanism under its stated backbone and protocol. A mask head still depends on successful detection.
- The standard-library reference checks sampling and loss arithmetic on authored inputs; it does not reproduce a trained detector, COCO accuracy or inference speed.
Why boxes need aligned spatial predictions
A bounding box can be a useful detection while still being a poor segmentation. Two people may have overlapping boxes, and neither rectangular region says which pixels belong to which person. Mask R-CNN attaches a small spatial predictor to each proposed object. Its central insight is that a mask needs the proposal's coordinate system to remain aligned with the features that predict it.
This review follows Mask R-CNN v3, the January 2018 revision of the ICCV 2017 paper. The publication year describes the original conference work; the separate date above is this review's date. Interactive diagrams and code below are authored numerical illustrations. They contain no trained COCO predictions and establish no measured inference speed.
One proposal, three different questions
A two-stage detector first proposes candidate regions, then classifies and refines them. Mask R-CNN keeps that organization and adds a mask branch in parallel with the classification and bounding-box branches. The outputs answer related questions: what category is this region, where is its box, and which locations within that region belong to the object? Original paper, §3
This makes instance identity explicit. If two detected people overlap, their masks live in different region predictions. The same image pixel can be evaluated within two different candidate regions. By contrast, a semantic segmentation map normally provides one category distribution at each image location without a separate identity for every object.
The region representation creates a convenient division of labor. The detection branch can aggregate evidence to identify a category; the mask branch retains a spatial arrangement for pixel prediction. A good object classifier does not automatically imply a good mask. Small coordinate errors that barely affect classification can move a predicted boundary enough to degrade mask overlap.
There are limits to this division. An object missed by the proposal and detection stages cannot receive a successful final instance mask. A poorly localized region can make the downstream task harder. Adding a mask head therefore does not make detection quality irrelevant, and masks should not be evaluated as if every ground-truth object were handed to the model in a perfect box.
Why quantizing a region is costly
Backbone features live on a coarse spatial grid. An image coordinate generally does not land exactly on a feature-grid center after division by the backbone stride. If a box edge is at x=23 pixels and the feature stride is sixteen, its projected coordinate is 1.4375. Rounding that number discards spatial information before the mask branch has seen it.
RoIPool performs quantization while turning regions of different sizes into fixed-size tensors. This is useful for recognition, but a mask predictor cares about the exact relationship between region coordinates and feature positions. Rounding the outer box and then rounding internal bin boundaries can introduce separate displacements. Original paper, RoIAlign discussion
The correction is conceptually small: retain floating-point coordinates for the region, bins and sample locations. The feature map still consists of discrete stored values. A continuous location is evaluated by interpolation, rather than by pretending that a learned feature vector was originally stored there.
Bilinear interpolation reads between centers
Within a grid cell, bilinear interpolation combines the values at four surrounding centers. The weights depend on horizontal and vertical fractional offsets. They are nonnegative for an in-bounds point and sum to one, so the interpolated scalar lies within the surrounding scalar values. Channels are processed independently using the same spatial weights.
Read a feature between grid centers
Bilinear interpolation reads four nearby feature values at the actual floating-point coordinate. The grid is an authored scalar feature map, f[y,x] = y² + x².
Four weights: 0.375, 0.125, 0.375, 0.125. Their sum is one.
f(1.25, 1.50) = 4.2500
The explorer uses a small authored map whose value at integer location (x,y) is x²+y². This is deliberately not a learned activation map. The result is a piecewise bilinear interpolation of those samples, not an evaluation of the continuous quadratic function at arbitrary coordinates. At an integer grid center, interpolation returns the stored value exactly.
An affine field provides a useful stronger test. If the stored feature is 2x+3y+1, bilinear interpolation reproduces that expression exactly everywhere inside the grid. That identity catches swapped axes and incorrect corner weights. Checking only integer coordinates would miss many interpolation bugs because the fractional weights vanish there.
Boundary behavior also belongs in the definition. This reference clamps coordinates to the final grid center, so a sample beyond an edge reads its boundary value. That is an explicit pedagogical choice. Production kernels may apply validity checks or other edge policies. Reading a source implementation requires checking those cases instead of assuming that the name “bilinear” completely specifies them.
Follow the actual sample coordinates
Divide a region into an output grid. For bin column c in an output of width W, and horizontal sub-sample index s among S samples, the reference places a sample at:
The y coordinate follows the corresponding row formula. With S=2, each bin contains four regularly spaced sample points. Interpolate each point, then aggregate the values in that bin. The paper considers both average and max aggregation. The example uses the average. Original paper, Figure 3 and §3
Keep the RoI and its sample points continuous
A 2×2 output uses four regularly spaced bilinear samples per bin, then averages them. Shift the box to see sampling move smoothly.
box = [0.50, 0.60, 3.50, 3.60]
bin (0,0)
3.950
first sample (0.875, 0.975)
bin (0,1)
9.950
first sample (2.375, 0.975)
bin (1,0)
10.350
first sample (0.875, 2.475)
bin (1,1)
16.350
first sample (2.375, 2.475)
Top-left bin: 3.9500 from 1.85 + 3.85 + 4.05 + 6.05 divided by four.
The moving box shows why actual locations matter. A fractional translation changes samples continuously. If the box is first rounded, a small movement can have no effect until a rounding threshold is crossed, followed by a jump. The rounded option intentionally demonstrates only that error source. It is not a complete implementation of RoIPool, which also has its own bin quantization and pooling rules.
The coordinate convention is explicit: integers identify feature-grid centers; image coordinates must first be projected into feature coordinates by their stride. We use continuous unaligned coordinates with no −0.5 shift, no legacy “plus one” width, and no forced minimum extent. Detectron2's aligned=True convention shifts coordinates by half a pixel. Different conventions can each be coherent, but mixing one convention's boxes with another convention's sampler changes the operation. Official Detectron2 RoIAlign documentation
A useful implementation review therefore begins with a tiny feature map and a few known fractional boxes. Record every sample coordinate. Test an exact integer center, a point immediately on either side of a grid boundary, a singleton spatial dimension and the final row or column. A visually plausible mask on a real photo is a much weaker diagnostic for a half-pixel error.
Classification selects the mask; masks do not classify each other
For a positive training region with ground-truth class k, the mask head predicts K binary masks but the mask objective supervises only channel k. Each pixel in that channel uses a sigmoid and binary cross entropy. The other class-specific mask channels do not contribute to that region's mask loss. Original paper, §3
Supervise only the matched class mask
A positive RoI has one ground-truth class and a binary mask. Its class selects one mask channel; the pixel loss does not make the K channels compete.
Target binary mask: [1, 0; 0, 1]. Class A logits stay [2, −1; −2, 1].
Class A (supervised)
Class B (ignored)
Mean mask BCE = 0.220095
With class A matched, changing class B has zero effect on this loss. Choosing B changes which channel receives supervision.
This differs from a per-pixel softmax over K categories. A softmax would make all categories compete at every pixel. In the original design, the classification branch decides which category the instance has, and the corresponding mask channel describes its spatial extent. The independent binary masks decouple those roles.
During training, the ground-truth class selects the supervised channel. During inference, the classification branch's predicted class selects the output channel. Confusing these two selection rules would either leak the answer at inference or supervise an arbitrary predicted channel during training. The binary target is created relative to the assigned instance, not the union of every same-category object that happens to intersect the region.
Negative regions contribute to classification but do not receive a foreground instance mask target. The combined training objective is:
The tiny example isolates the last term. Its mean binary cross entropy uses a stable logits expression, so confidently correct logits of magnitude one thousand do not require taking the logarithm of a rounded probability. Changing an unmatched channel has no effect on this loss. That is a semantic property of the objective, rather than a numerical accident.
What the experiments isolate
The paper trains on COCO trainval35k, combining 80k training images with 35k validation images, and uses the remaining 5k minival images for ablations. In the ResNet-50-C4 alignment comparison, Table 2(c) reports mask AP 26.9 for RoIPool, 30.2 for max-aggregated RoIAlign, and 30.3 for average-aggregated RoIAlign. The corresponding AP75 values are 26.4, 31.8 and 31.5. These are original reported measurements, not results of this page's reference. Original paper, §4 and Table 2(c)
This comparison is valuable because it probes the alignment mechanism within the same backbone setting. The table also includes bilinear RoIWarp without correct alignment, which does not yield the same gain. Interpolation by itself is therefore not an adequate description of the contribution; the coordinates at which it is performed are central.
These AP values are averaged over mask IoU thresholds under the COCO protocol. They are not pixel accuracy and should not be compared numerically with a semantic segmentation IoU from another dataset. Table 1 uses test-dev for main comparisons, a different evaluation split from the ablations. Keeping the backbone, split, aggregation and metric attached to each number prevents a table assembled from several settings from becoming misleading.
The evidence is also bounded by the task and recipe. The experiments support instance segmentation under the stated detector and training conditions. They do not establish that a supplied toy feature grid reconstructs real objects, that a mask head can recover missed detections, or that all framework coordinate conventions yield identical outputs. A deployment must still evaluate small objects, occlusion, annotation quality and domain shift in its own data.
Inspect the source and run the arithmetic
Each row below couples a relevant original PDF crop to its mathematical interpretation and a synchronized code region. The manifest records the source revision, PDF hash and exact crop geometry. These crops are source evidence, while the diagrams above are authored illustrations.
1. Evaluate features at floating-point positions
Original paper
Figure 3 depicts four regularly placed samples in each bin.
Math and tensor shapes
Integer coordinates denote feature-grid centers. The reference clamps at the grid boundary and performs no half-pixel shift. Bilinear interpolation exactly reproduces affine fields; fractional coordinates need not correspond to stored pixels.
Python implementation
def bilinear(values, height, width, x, y):
"""Interpolate four grid centers with an explicit clamped boundary policy."""
if height <= 0 or width <= 0 or len(values) != height*width:
raise ValueError("Invalid feature map")
if not all(math.isfinite(v) for v in [x, y, *values]):
raise ValueError("Values and coordinates must be finite")
x, y = max(0, min(width-1, x)), max(0, min(height-1, y))
x0, y0 = math.floor(x), math.floor(y)
x1, y1 = min(x0+1, width-1), min(y0+1, height-1)
dx, dy = x-x0, y-y0
return ((1-dy)*((1-dx)*values[y0*width+x0]+dx*values[y0*width+x1])
+ dy*((1-dx)*values[y1*width+x0]+dx*values[y1*width+x1]))2. Sample inside bins without rounding the RoI
Original paper
The RoIAlign description removes quantization of the RoI, bins and sample coordinates.
Math and tensor shapes
Use the analogous y equation. S=2 gives four locations per bin; this reference averages their interpolated values. The paper allows average or max aggregation. Image boxes must first be scaled to feature coordinates. Framework aligned modes and legacy minimum extents need separate explicit conventions.
Python implementation
def roi_align(values, height, width, box, out_height=2, out_width=2, samples=2):
"""Four regular sample locations per bin, then average their values."""
if len(box) != 4 or not all(math.isfinite(v) for v in box):
raise ValueError("Invalid box")
x1, y1, x2, y2 = box
if x2 <= x1 or y2 <= y1 or min(out_height, out_width, samples) <= 0:
raise ValueError("Box and output sizes must be positive")
output = []
for row in range(out_height):
for col in range(out_width):
points = []
for sy in range(samples):
for sx in range(samples):
x = x1+(col+(sx+0.5)/samples)*(x2-x1)/out_width
y = y1+(row+(sy+0.5)/samples)*(y2-y1)/out_height
points.append(bilinear(values, height, width, x, y))
output.append(sum(points)/len(points))
return output3. Select the ground-truth class before mask BCE
Original paper
Section 3 defines the multitask objective and the matched-class binary mask loss.
Math and tensor shapes
Only a positive RoI contributes mask loss, using its ground-truth class k. Other mask channels are ignored by this RoI loss. The stable logits expression avoids taking log(0); it is independent sigmoid BCE, not a pixelwise softmax over object classes.
Python implementation
def mask_loss(logits_by_class, target, class_index):
"""Mean independent sigmoid BCE on the ground-truth class channel only."""
if not 0 <= class_index < len(logits_by_class) or not target:
raise ValueError("Invalid matched class or empty target")
if any(t not in (0, 1) for t in target):
raise ValueError("Masks must be binary")
if any(len(row) != len(target) for row in logits_by_class):
raise ValueError("Mask shape mismatch")
logits = logits_by_class[class_index]
if not all(math.isfinite(z) for row in logits_by_class for z in row):
raise ValueError("Logits must be finite")
return sum(max(z, 0)-z*t+math.log1p(math.exp(-abs(z)))
for z, t in zip(logits, target))/len(target)Original PDF excerpts: Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. Source 1703.06870v3; version, crop rectangles and hashes are recorded in the manifest.
Run python examples/maskrcnn.py --json, or download the standard-library reference. The checks cover affine interpolation, exact integer coordinates, clamped boundaries, continuous bin sampling, extreme logits and independence from unmatched class channels. See the README and requirements for the coordinate contract and scope.
The reference leaves out the learned backbone, region proposal network, box predictor, mask convolutions, training assignment and suppression pipeline. This is deliberate: those pieces are necessary for an actual detector, while the small reference makes the alignment and loss claims directly testable. It should be used to understand and debug a mechanism, not presented as a reproduced COCO system.
Related reading
- RoI pooling provides the fixed-size region extraction background.
- Image segmentation distinguishes semantic and instance targets.
- Bounding-box regression explains the parallel localization task.
- Feature Pyramid Networks supplies semantically enriched region features at several scales.
- U-Net takes a different route to dense spatial predictions.
