RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

What does a detector actually return?

An image contains pixels. An object detector must turn them into a variable-sized collection of boxes, class labels and scores. A box says where an object might be; a label says which category the detector assigns to it; a score helps rank and filter competing predictions. Before examining architecture search, start with those observable outputs.

Choose a photograph below, inspect a saved detection, then change the threshold. A query is an internal output slot that can propose a box and class scores. Its index is not an object identity that persists across images. Two class predictions may even refer to the same query box.

The explorer's provenance identifies the recorded software, checkpoint, source images and preprocessing. Its controls filter saved candidates; they do not execute a network in your browser. A lower threshold cannot recover a query/class pair discarded by the export's original top-K. Scores are sigmoid detection scores, without a demonstrated calibration guarantee. Two photographs illustrate behavior; they establish neither COCO AP nor robustness nor inference speed.

This review follows paper version 2, dated February 3, 2026. The first preprint appeared in November 2025; the publication venue is ICLR 2026. The date above is the date of this write-up. Throughout, paper measurements, recorded pretrained output, and authored teaching calculations have separate evidence. A released package is not proof that the paper's original training or benchmark was reproduced.

The paper's central idea is to adapt a shared detector to a dataset and then select an architecture for a latency budget. To understand why that helps, first follow how one image becomes those candidate objects.

Follow one object through the network

Step from the image grid to the object slots. The architecture figure tracks two kinds of representation: features indexed by image position, and features indexed by query. Spatial features preserve where evidence occurs; query features gather evidence needed to predict an instance.

Section 3 starts from LW-DETR and replaces its CAEv2 backbone with pretrained DINOv2. The backbone interleaves local window attention with global attention. Window attention limits which tokens interact in a block; global blocks let information cross those local boundaries. The multi-scale projector prepares spatial features for the detector and uses layer normalization, making its normalization independent of batch statistics during gradient accumulation.

Decoder queries read the projected image features through deformable cross-attention. This inherited mechanism samples selected spatial locations rather than forming a dense query-by-every-pixel interaction. Any sampled-point paths in the teaching trace are schematic: they are not measured attention weights, learned offsets from the recorded Nano run, or an explanation of why a particular saved box was predicted.

The original architecture diagram makes the shared spatial organization explicit. Detection and segmentation both use interpolated projector features, while query embeddings feed the prediction branches.

Enlarge figure

RF-DETR does not originate set prediction, DINOv2 pretraining, deformable attention, auxiliary decoder supervision or query/pixel mask products. Its contribution combines a foundation-backed real-time detector with end-to-end weight-sharing architecture search, a revised adaptation/training recipe, and a more controlled latency evaluation. The authors' priority claim concerns that search setting; it should not be read as inventing its inherited components.

That distinction suggests the next question. If the same components can be arranged with different spatial and object budgets, why deploy one fixed arrangement on every dataset?

Turn one trained network into several operating points

Change one workbench control at a time. Resolution and patch size alter the spatial sequence; window count changes local grouping; query count and decoder depth alter the object-processing branch. The displayed counts are calculated geometry, not a prediction of AP or milliseconds.

For a chosen image of height H and width W, divisible by square patch size p, the backbone receives:

At 384 × 384 with patch size 16, this gives 24 × 24 = 576 tokens. Doubling both image dimensions at the same patch size quadruples the token count. A global attention matrix then has sixteen times as many token pairs. That is a calculation about one operation's dimensions, not a sixteen-fold end-to-end latency claim: projections, local attention, memory movement and decoder work scale differently.

For nondivisible dimensions, a drawing must specify whether it resizes, pads or rejects the shape. The downloadable reference uses explicit right/bottom padding and ceiling division. The workbench states its own geometry. Neither implements the paper's FlexiVIT-style patch-weight adaptation merely by resizing a grid. Changes in patch size require transforming learned patch weights; positional embeddings must also adapt to the spatial layout.

The paper's five knobs are shown together below. Its broad description of randomly sampling architectures needs the qualification in Appendix A: training fixes six decoder layers and 300 queries. Resolution, patch size and windows vary during shared training; decoder layers and queries are dropped at inference.

Enlarge figure

Weight sharing avoids separately training every candidate from scratch. It does not make searching free. Appendix A reports shared training taking roughly two to four times a non-NAS baseline and estimates approximately 10,000 GPU-hours for search. It also contains a counting inconsistency: the reported 6,468 configurations use seven patch sizes, while the listed inference grid includes eight, adding size 14. The reported total should not be silently replaced by a recomputed product.

The benefit is a family of operating points from shared weights. The unresolved question is how those weights become useful across both architecture choices and target domains.

Train once, adapt, search, optionally refine

Follow the timeline in order. Foundation initialization supplies image representations, detection or segmentation pretraining teaches task heads, and supervised target training adapts shared weights. Validation search then chooses a configuration; optional further fine-tuning updates the chosen subnet. Selection and learning have different roles even when both use target-domain data.

The paper uses Objects-365 pretraining and, for segmentation, SAM2 pseudo-masks on that dataset. Pseudo-labels make mask supervision available but are model-generated targets rather than human mask ground truth. Target adaptation is supervised specialization. DINOv2's broad pretraining does not turn this into a zero-shot open-vocabulary detector.

The paper training recipe uses learning rate 10⁻⁴, batch size 128, per-layer learning-rate decay 0.8 and gradient clipping at 0.1, without learning-rate warm-up. It limits augmentation to horizontal flips and random crops, and resizes at batch level to reduce padding artifacts and wasted work. “Scheduler-free” needs qualification: the EMA still has a schedule. These are paper settings, not a claim about every current trainer's defaults.

Matching teaches slots to cover different objects. If there are T labeled objects and Q\ge T candidate queries, an injective assignment sends each target to a distinct query:

Here C is a matching cost matrix. Inherited DETR-style matching uses classification and box geometry to decide correspondence. The discrete matching cost is not the differentiable training loss. After correspondence is chosen, the training criterion supervises class and box predictions, and masks when present. The tiny reference accepts a supplied cost matrix; it does not implement RF-DETR's official cost coefficients, focal classification, box losses, grouped training or optimizer.

Intermediate decoder outputs receive auxiliary supervision, so later dropping a decoder suffix does not expose a completely unsupervised prediction head. That supports selectable exits, but it does not prove every exit has equal quality. Validation must still determine whether an exit fits the task.

Now the architecture budget can be interpreted concretely: which object slots survive, and how much refinement does each receive?

Spend queries and decoder layers deliberately

Reduce the query budget, then inspect a shallower decoder exit. Include the zero-layer exit: the network still has its backbone, projector and encoder proposals, even when no decoder block executes. “Zero decoder layers” does not mean zero inference work.

Appendix B ranks encoder proposals by the maximum sigmoid class score of each token, then retains a query budget. Those retained queries receive the selected number of decoder refinements. A budget of 50 cannot supply 100 distinct query boxes for 100 different instances; sorting cannot restore a discarded proposal. This makes crowding and small-object coverage important validation cases rather than conclusions that can be read from a parameter count.

There is a second ranking after the final prediction head. The released detection postprocessor applies independent sigmoids to class logits and ranks the flattened query/class pairs. A single query can therefore contribute several class candidates with the same box. That differs both from encoder query-budget ranking and from original DETR's softmax with an explicit no-object category.

In this notation r_q illustrates per-query ranking; \mathcal I_K denotes final pair ranking. They act at different stages and need not use the same logits. The pinned 1.11.2 postprocessor caps K to available pairs and uses stable descending sorting, preserving flattened-index order on ties. Its box-only path returns selected candidates without threshold filtering. The detection caller subsequently keeps scores strictly greater than its threshold. Any recorded 1.9.4 run is independently bound to its exporter/source receipts, rather than inferred from this later source snapshot.

No extra NMS is added to this plain detection explanation. End-to-end training encourages distinct objects, while the output contract still permits repeated query boxes across class pairs. A threshold slider filters that contract; it cannot convert scores into calibrated probabilities or solve missed-object recall.

Queries describe instances. To give those instances a shape more detailed than a rectangle, the next branch reconnects them to spatial features.

Give every query a mask

Inspect a query vector and one pixel embedding, then compare their dot product with its sigmoid. The spatial grid below is deliberately tiny so every multiplication can be checked. All displayed embeddings are authored; no RF-DETR segmentation checkpoint generated them.

Section 3 and Figure 2 add a lightweight segmentation head inspired by prior segmentation work, including MaskDINO. A learned projector creates a spatial pixel embedding map; an FFN projects each decoder query into the same feature dimension. Their dot product yields one mask logit at each pixel:

A logit is the unbounded dot product. A sigmoid score lies between zero and one. A binary mask requires an additional threshold and a stated spatial resizing policy. Keeping these separate prevents a zero logit from being misread as zero foreground score: its sigmoid is 0.5.

For a projected query [1, 2], pixel vectors [1, 0], [0, 1], [-1, 1] and [2, -1] give logits 1, 2, 1, 0. This checks the output arithmetic only. It says nothing about how well a trained projector separates a cat from its background.

RF-DETR shares interpolated spatial features between its detection and segmentation paths and omits a multi-scale backbone fusion in the mask branch to reduce latency. Mask supervision at decoder layers also makes decoder truncation relevant to segmentation. These are architectural and training choices; the basic query/pixel product and prototype interpretation have predecessors.

The mechanics are now small enough to test directly. The next section connects the original source, the equations and one downloadable program, so an attractive diagram is not the only evidence of correspondence.

Make the explanation executable

The five rows below preserve the boundaries established above: chosen-grid accounting, inherited assignment, released-software selection, mask arithmetic and task-specific Pareto filtering. Each excerpt is generated from the same standard-library Python file offered for download. The full source includes validation helpers that are outside individual excerpts.

Authored CPU calculations, using only Python’s standard library. Each excerpt comes from the same downloadable source. Paper references explain the relationship; they do not make these helpers an official model.

Reference fixtureStandard tiny examplePadding, ties and equality

Authored inputs and independently tested expected outputs. Selecting a fixture displays a test case; it does not run Python in the browser.

1. Count a chosen patch grid before estimating work

Original paper

Enlarge figure

Math and tensor shapes

R=\lceil H/p\rceil,\quad C=\lceil W/p\rceil,\quad N=RC

H_{\rm padded}=Rp,\quad W_{\rm padded}=Cp

Exact accounting for the explicitly chosen right/bottom padding convention. An authored 17×33 input and patch 16 give a 2×3 grid padded to 32×48. This is neither the official square resize nor a claim that arbitrary padded shapes are valid upstream NAS configurations. Token count alone predicts no AP or milliseconds.

patch_geometry(384, 384, 16)

Expected: 24 rows × 24 columns = 576 tokens; padded size 384×384

H=W=384, p=16 → 24×24=576 tokens; no padding

Python implementation

python

def patch_geometry(height, width, patch):
    """Count a chosen square-patch grid with explicit right/bottom padding.

    This is bookkeeping, not RF-DETR's official image resize or FlexiVIT
    patch-weight transformation. Divisible inputs require no padding.
    """
    for value in (height, width, patch):
        _positive_integer(value)
    rows = (height + patch - 1) // patch
    columns = (width + patch - 1) // patch
    return {
        "rows": rows,
        "columns": columns,
        "tokens": rows * columns,
        "padded_height": rows * patch,
        "padded_width": columns * patch,
    }

2. Find a global assignment, then distinguish the learning loss

Original paper

Enlarge figure

Math and tensor shapes

\hat\pi=\arg\min_{\pi\ {\rm injective}}\sum_{t=1}^{T} C_{\pi(t),t}

With rows [1,2], [2,100], [9,9], the global optimum assigns target 0 to query 1 and target 1 to query 0. Greedily taking cost 1 first would force an expensive second match. This exhaustive solver is bounded to three targets and twelve queries. It does not compute focal, L1 or GIoU costs, gradients, grouped training or the weighted training loss. A target is a column, not a class slot.

minimum_assignment([[1,2],[2,100],[9,9]])

Expected: queries [1,0]; cost 4

3 queries × 2 targets → query indices [1,0], total cost 4

Pinned official matcher.py (1.11.2 source snapshot)

Python implementation

python

def minimum_assignment(costs):
    """Find the minimum injective assignment for at most three targets.

    costs[query][target] is an already supplied TEACHING cost. It is not the
    official focal/L1/GIoU matcher cost or the differentiable training loss.
    Exhaustive search illustrates the optimum a Hungarian solver would find.
    Equal totals choose the lexicographically first query tuple.
    """
    if not costs:
        return {"queries": [], "cost": 0}
    if all(not row for row in costs):
        return {"queries": [], "cost": 0}
    _matrix(costs)
    targets = len(costs[0])
    if targets > 3 or len(costs) > 12 or targets > len(costs):
        raise ValueError("teaching solver needs <=3 targets and <=12 queries")
    choices = itertools.permutations(range(len(costs)), targets)
    best = min(choices, key=lambda qs: (sum(
        costs[q][t] for t, q in enumerate(qs)), qs))
    return {
        "queries": list(best),
        "cost": sum(costs[q][t] for t, q in enumerate(best)),
    }

3. Preserve query/class pairs through sigmoid and top-K

Original paper

Enlarge figure

Math and tensor shapes

s_{qc}=\sigma(\ell_{qc}),\quad (q,c)\in\operatorname{TopK}_{q,c}(s)

\mathcal D_\tau=\{(q,c,s_{qc}): (q,c)\in\operatorname{TopK}(s),\ s_{qc}>\tau\}

Logits log(4), log(3) and 0 become scores 0.8, approximately 0.75 and 0.5. At threshold 0.5, equality is excluded. Two high classes from one query may both survive and share its box; no NMS is added. Original class-slot indices remain intact. Stable ties preserve flattened-index order and K caps to available pairs, matching pinned 1.11.2. Its box-only PostProcess returns selected candidates without thresholding; strict thresholding is applied by the detection caller. Any 1.9.4 export is qualified separately. The browser filters an already recorded official candidate list.

select_pairs([[log(4),log(3)],[0,-1]], 3, 0.5)

Expected: pairs (query 0, class 0) score 0.8 and (query 0, class 1) score ≈0.75

logits [queries, class slots] → K pairs → strict threshold

Pinned official postprocess.py (1.11.2 source snapshot)

Python implementation

python

def sigmoid(logit):
    """Compute a stable scalar sigmoid; scores are not calibrated confidence."""
    if not math.isfinite(logit):
        raise ValueError("logit must be finite")
    if logit >= 0:
        return 1 / (1 + math.exp(-logit))
    exponential = math.exp(logit)
    return exponential / (1 + exponential)


def select_pairs(logits, top_k, threshold):
    """Rank flattened query/class pairs, then apply a STRICT score threshold.

    Class slots retain their original indices, including vocabulary gaps.
    Different classes may reuse the same query box. No NMS or softmax.
    Stable ties preserve flattened index; K caps to available pairs, as in
    the pinned 1.11.2 postprocessor. Strict threshold is the caller filter,
    not the box-only PostProcess return path.
    This accepts logits, not the saved already-ranked official candidate list.
    """
    _matrix(logits)
    if type(top_k) is not int or top_k < 0:
        raise ValueError("top_k must be a nonnegative integer")
    if not math.isfinite(threshold) or not 0 <= threshold <= 1:
        raise ValueError("threshold must lie in [0, 1]")
    candidates = [
        {"query": q, "class_slot": c, "score": sigmoid(logit)}
        for q, row in enumerate(logits) for c, logit in enumerate(row)
    ]
    candidates.sort(key=lambda item: (-item["score"], item["query"],
                                      item["class_slot"]))
    return [item for item in candidates[:top_k]
            if item["score"] > threshold]

4. Keep mask logits distinct from sigmoid display scores

Original paper

Enlarge figure

Math and tensor shapes

M_{qhw}=\sum_{d=1}^{D}Q_{qd}P_{hwd},\quad S_{qhw}=\sigma(M_{qhw})

For query [1,2], pixel vectors [1,0], [0,1], [-1,1] and [2,-1] produce logits [1,2;1,0]. The last value is zero, corresponding to sigmoid score 0.5. The function returns logits and performs no threshold. It receives already projected features and omits interpolation, the learned FFN, training losses and checkpoint inference. These are authored arithmetic values, not measured RF-DETR masks.

mask_logits([1,2], [[[1,0],[0,1]],[[-1,1],[2,-1]]])

Expected: logits [[1,2],[1,0]]

query [2], pixels [2,2,2] → logits [2,2]

Python implementation

python

def mask_logits(query, pixels):
    """Return query-pixel dot products on an authored H x W x D embedding.

    query is already projected into the pixel feature dimension. Results
    are logits, not masks from a checkpoint. Apply sigmoid explicitly for
    display scores; threshold/interpolation and training losses are separate.
    """
    if not query or any(not math.isfinite(x) for x in query):
        raise ValueError("query must be a finite nonempty vector")
    if not pixels or not pixels[0]:
        raise ValueError("pixel grid must be nonempty")
    width = len(pixels[0])
    if any(len(row) != width for row in pixels):
        raise ValueError("pixel grid must be rectangular")
    vectors = [vector for row in pixels for vector in row]
    if any(len(vector) != len(query) for vector in vectors):
        raise ValueError("query and pixel dimensions must agree")
    _matrix(vectors)
    return [[sum(q * p for q, p in zip(query, pixel)) for pixel in row]
            for row in pixels]

5. Compare operating points only within one task and protocol

Original paper

Enlarge figure

Math and tensor shapes

a\prec b\iff t_a\le t_b\land A_a\ge A_b\land(t_a<t_b\lor A_a>A_b)

These invented (milliseconds, AP) pairs test dominance only. Equal points remain; different task/protocol identifiers never dominate each other. Real protocol identifiers must encode dataset/split, resolution policy, hardware, precision, runtime, batch and timing scope. The helper does not authenticate those conditions or implement upstream weight-sharing NAS. It never interpolates an empirical frontier from sparse results.

same-protocol points a=(2,40), b=(3,39), c=(4,45)

Expected: retain a and c; b is dominated

authored (2,40), (3,39), (4,45) → retain first and third

Python implementation

python

def pareto_frontier(points):
    """Keep nondominated authored points, only comparing like task/protocol.

    Lower latency and higher AP are preferred; one inequality must be strict.
    Equal points remain. Protocol must encode split, hardware, precision,
    batch, runtime and timing scope before real measurements are compared.
    This is exhaustive filtering, not the paper's NAS training algorithm.
    """
    for point in points:
        if not point.get("task") or not point.get("protocol"):
            raise ValueError("task and protocol must be explicit")
        if (not math.isfinite(point["latency_ms"])
                or point["latency_ms"] <= 0
                or not math.isfinite(point["ap"])
                or not 0 <= point["ap"] <= 100):
            raise ValueError("need positive finite latency and AP in [0, 100]")

    def dominates(left, right):
        same = (left["task"], left["protocol"]) == (
            right["task"], right["protocol"])
        no_worse = (left["latency_ms"] <= right["latency_ms"]
                    and left["ap"] >= right["ap"])
        better = (left["latency_ms"] < right["latency_ms"]
                  or left["ap"] > right["ap"])
        return same and no_worse and better

    return [point.copy() for point in points
            if not any(dominates(other, point) for other in points)]

Run offline with Python 3.9+. No packages, weights or network access are required. Smoke prints deterministic JSON for all five regions. Copy the complete file to include the validation helpers outside the excerpts. This performs no training, attention export or benchmark evaluation.

rfdetr.pyrequirements-rfdetr.txt

python rfdetr.py --smoke

The program runs offline without model weights or framework installation. Its deterministic smoke output verifies 576 tokens, assignment cost 4, two surviving class pairs from one query, the mask logits above, and an authored dominance example. These checks establish exact toy arithmetic. They do not establish the official training objective, a pretrained forward pass, NAS quality, AP or latency.

The recorded explorer has a separate evidence path: image and preprocessing tensors, complete relevant predictions, checkpoint/source identities and independent replay. Matching a toy sigmoid is useful, but it cannot authenticate a model run. Conversely, a real box overlay cannot demonstrate that an authored architecture animation depicts the executed tensors.

Correct mechanics explain what a result means. The final step is to decide which measured results can fairly be compared.

Read an accuracy–latency claim with its conditions

Switch the evidence board between detection, segmentation and transfer. Read the conditions beside a point before comparing coordinates. Box AP and mask AP answer different questions, and an RF100-VL average over 100 datasets is not COCO validation AP.

In the paper's COCO validation results, detection Nano reaches 48.0 box AP versus 42.7 for D-FINE Nano, detection 2XL reaches 60.1 box AP, and segmentation Nano reaches 40.3 mask AP. These are AP50:95 figures, with reported latencies of 2.3, 2.1, 17.2 and 3.4 ms respectively under the paper's T4/TensorRT 10.4/CUDA 12.4 protocol; they are not measurements of this website's recorded export. Consult Tables 2 and 3 and the board's per-row conditions for resolution, precision, batch and timing scope before making a comparison.

The paper experiments use AP50:95 and report latency on an NVIDIA T4 with TensorRT 10.4 and CUDA 12.4; starred comparators use PyTorch. Precision and the actual exported artifact matter. The authors show that changing export or quantization can change accuracy, so FP32 accuracy paired with a different FP16 timing artifact is not a faithful operating point. Missing per-row conditions belong in the evidence card as missing, rather than being filled from a current package README.

The protocol inserts 200 ms pauses between forward passes and uses CUDA graphs to stabilize measurement. Those pauses address thermal and power-state variation; they do not measure sustained throughput. Taking the reciprocal of a buffered latency and calling it an application frame rate would omit preprocessing, transfers, queues and any other work outside the timed scope.

For detection comparisons, NMS belongs in timing when it is required to obtain the reported outputs. For segmentation, generating prototype features is not the same amount of work as returning usable per-instance masks. Input resolution, batch, precision, backend and output construction must travel with a latency number. A “Nano” label alone is insufficient: the paper groups models by latency buckets, not equal parameter count.

Transfer results require another distinction. RF100-VL selects dataset-specific subnetworks near a target latency. The paper separately reports optional additional fine-tuning after selection; its 2XL average rises from 63.3 to 63.5 AP, while the table reports 15.6 ms average latency. The Table 4 conditions and dataset-specific search belong to that claim. This is supervised transfer, not an unseen-label evaluation. A mixed TensorRT/PyTorch comparison may describe the reported systems, but it cannot isolate architecture as the cause of their speed difference.

The practical outcome is a decision process. Start with a representative target-domain split, choose a latency scope that matches deployment, compare actual exported artifacts, and validate a query/depth budget against difficult scenes. RF-DETR provides a way to specialize shared weights across such operating points. Whether that yields the best detector for a new application remains an empirical question, not a consequence of the search-space drawing.

Continue from here

The preceding chapters explain Deformable DETR’s sparse sampling, DINO detector’s query training, and LW-DETR’s lightweight architecture. Detector DINO is distinct from the self-supervised DINOv2 backbone used here.

DETR explains the inherited set-prediction problem and why matching matters. Modern object detection places query detectors beside other families; cross-attention explains how object slots read image features. For spatial computation, continue with ViT self-attention, feature pyramids and layer normalization. NMS and Soft-NMS clarify the postprocessing distinction that must remain visible in comparisons.

Primary references are the version-pinned paper, its ICLR proceedings record, the official repository, the 1.11.2 source snapshot, and the explorer's separately qualified recorded-output receipts. The original Figures 2 and 3 retain their attribution and CC BY 4.0 license above. Neither those figures nor this CPU reference claims reproduction of the paper benchmarks.