Skip to main content

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Trace Grounding DINO through text-image enhancement, language-guided anchors, cross-modal decoding and phrase grounding, with Python and evaluation caveats.

TL;DR

  • Grounding DINO lets language influence feature enhancement, anchor selection and decoder refinement. It belongs to the detection DINO lineage, separate from self-supervised DINO/DINOv2.
  • Language-guided selection ranks image positions by their best token compatibility, initializing spatial anchors while retaining learned query content. Sub-sentence masks preserve phrase-level text structure.
  • Outputs pair boxes with token alignments. Box retention and phrase decoding use separate thresholds; a retained box need not have a useful or even nonempty decoded phrase.
  • Open-set prompts do not establish unseen-concept generalization. Reported detection and referring-expression results depend on checkpoint, training sources, splits and metrics, including possible image overlap.
  • The deterministic explorers and Python reference explain mechanisms rather than measured detector behavior. For an application, test box quality and phrase correctness separately under recorded prompts and preprocessing.

A detector should know what you are asking it to find

“Cup” and “the red cup beside the bowl” ask different things of a detector. The first requests a category. The second asks it to resolve attributes and relationships in a scene. A conventional closed-set detector predicts coordinates and scores over a fixed training vocabulary. Replacing its final classifier with text embeddings adds flexibility, but leaves a deeper question: at what point should the words influence the search for the object?

Grounding DINO's answer is to introduce language throughout that search. Image and text features interact before proposals are selected. Language then helps select the spatial references from which queries start. Finally, each decoder query reads both image and text features as it refines a box. The output is a set of boxes with token alignments, from which phrases can be decoded. This three-part design is the organizing idea of the paper's Figure 3 and §3.

This is the detection DINO lineage, not the similarly named self-supervised DINO/DINOv2 image encoder. The preprint appeared in March 2023; the reviewed revision is July 2024 v5, and the publication venue is ECCV 2024. Publication metadata uses the conference year; the review date above is separate.

All six explorers below use authored illustrations. Feature products, attention weights and box updates are calculated from those authored inputs. Benchmark values are separately labeled cited measurements. There are no recorded checkpoint predictions or new runtime measurements on this page.

First make the image and the words aware of each other

An image backbone produces multi-scale spatial features. A text backbone produces token features. Those streams initially have different roles: one describes where visual evidence occurs; the other describes the requested concepts and their context. Grounding DINO uses a feature enhancer to let them exchange information rather than waiting until a final scoring layer.

Within each modality, the image branch uses deformable self-attention and the language branch uses text self-attention. Across modalities, bidirectional attention allows image positions to read language and text tokens to read visual evidence. The enhanced representations therefore depend on the supplied image and the supplied prompt. See §3.1 and the pinned fusion implementation.

1. Let each modality read the other

Inspect one head on six authored spatial features and two text features. The same compatibility matrix is normalized across a different axis in each direction.

Wide diagram: swipe, or focus here and use the left and right arrow keys.

Image to text attention, selected query 0P0[2,0]P1[1.4,0.2]P2[0.1,1.8]P3[0.2,0.9]P4[0.1,0.1]P5[0.7,0.5]cup [1,0]0.804bowl [0,1]0.196Image features → compatibility ← text features
Row-normalized image → text weights
From / tocupbowl
P00.8040.196
P10.7000.300
P20.2310.769
P30.3790.621
P40.5000.500
P50.5350.465

Selected pooled value: [0.804, 0.196]. Weights sum to 1.000 over text keys.

Calculated one-head attention on supplied projected features. This omits learned projections, layer scales, residuals, normalization and the enhancer’s image/text self-attention. Patch locations are illustrative, not a learned heatmap.

The figure exposes a normalization detail that a diagram of two arrows can hide. If a compatibility matrix has image positions as rows and text tokens as columns, an image query normalizes across text keys. The reverse direction must normalize across image positions. Transposing an already normalized matrix does not produce the reverse attention distribution. It changes which values sum to one.

For one head on supplied projected features, the conceptual pooling is

A=Q_IK_T^\top/\sqrt{d_h},\qquad U_I=\operatorname{softmax}_{T}(A)V_T,\qquad U_T=\operatorname{softmax}_{I}(A^\top)V_I.

The explorer uses identity projections so every number can be inspected. Actual fusion has learned projections, numerical stabilization, residual paths and learned layer scales. It is only one part of a larger enhancer, not the entire encoder. The published default uses six enhancer layers, six decoder layers, a feature width of 256 and a maximum text length of 256 tokens; §4.1 specifies the configuration.

Early fusion has a practical consequence. A CLIP-style image representation can serve many text retrieval queries after one independent image encoding. Here, the fused representation changes with the prompt. Caching a backbone output is different from caching the entire language-conditioned encoder result. The flexibility of deep interaction brings prompt-dependent computation; the paper's architecture alone does not establish its deployment latency.

Then let language choose where queries begin

DETR-style decoding starts from object queries. Grounding DINO uses image-language compatibility to choose promising encoder positions before decoding. Let enhanced image features be I\in\mathbb R^{N_I\times d} and enhanced text features be T\in\mathbb R^{N_T\times d}. Its selection rule is

s_i=\max_t(I_i^\top T_t),\qquad J=\operatorname{TopK}_i(s_i).

The reduction order matters: first take the best text-token score for each spatial position, then rank positions. This neither selects one query per word nor globally ranks every image/token pair. Several selected positions can align most strongly to the same token, and one phrase can receive none. Equation 1 and §3.2 describe the selection.

2. Change the prompt before selecting anchors

Rank spatial positions by their largest valid-token logit. Multiple positions may select the same word; this is not a per-word quota or non-maximum suppression.

2

Wide diagram: swipe, or focus here and use the left and right arrow keys.

Selected spatial anchors 0, 2P0max 2.0P1max 1.4P2max 1.8P3max 0.9P4max 0.1P5max 0.7gather boxesPosition: selectedContent: learned

Ranked IDs: P0 (2.0) → P2 (1.8). Even with only “cup”, unrelated positions can fill K slots; top-K is not a relevance threshold.

Authored 2 × 3 feature grid; exact dot products. The real model uses 900 queries by default, not six. Stable index ordering for ties is a teaching convention, not a PyTorch topk guarantee.

Try “cup.” with K=2, then “bowl.”. The spatial ordering changes even though the authored image features stay fixed. Increasing K also admits weaker positions. Top-K always spends its available budget; it is not a confidence threshold. Nor does it suppress overlapping boxes as NMS would.

The distinction between query position and query content is essential. Following DINO's mixed query selection, the selected encoder predictions initialize dynamic anchor boxes, while content embeddings are learned parameters. “Language-guided queries” should not be read as copying each word vector directly into a query slot. The released transformer's gather and initialization path makes the separation visible.

The real default query count is 900. The six-position browser grid isolates the ranking rule. It does not estimate recall, learned objectness or query diversity. The selection dot products also have a different purpose from the attention softmax in the previous figure: ranking logits need not sum to one across words.

A selected anchor still needs to read both modalities

A proposal is a starting hypothesis, not a finished grounding result. Decoder self-attention lets queries interact; cross-attention lets each query read the enhanced language and image representations. Image reads use multi-scale deformable attention, preserving the sparse spatial sampling structure of the detection backbone. Text reads use dense attention over the enhanced language sequence.

There is a small but relevant paper/code distinction. §3.3 lists self-attention, image cross-attention, text cross-attention and an FFN. The pinned released forward method executes self-attention, text, image, then FFN. The explorer follows the implementation order; it does not assert that exchanging these operations is equivalent.

3. A query reads language and space separately

Select a layer operation. This ordering follows the pinned released transformer.py: self-attention → text → image → FFN. The paper’s prose lists image and text in the opposite order.

Wide diagram: swipe, or focus here and use the left and right arrow keys.

Decoder operation: Query self-attentionOther queriesself-attentionEnhanced textdense readsImage pyramidsparse readsBox + tokenheadsLearned query content + selected positional referenceEach attention/FFN sublayer also has residual and normalization paths.

Queries exchange information with other queries; this does not itself pick a text label.

0

Wide diagram: swipe, or focus here and use the left and right arrow keys.

Refined center 0.400, original center 0.4Dashed: reference. Solid: logit-space refinement. The display may extend beyond the image.

Refined (cx, cy, w, h): [0.400, 0.500, 0.300, 0.250]. A zero residual preserves the reference; a residual is not a pixel displacement.

Authored dataflow diagram. Moving the residual computes only the box-head arithmetic, not attention or a decoder forward pass. The real decoder repeats its layer six times and updates all four box coordinates.

After feature updates, a box head predicts offsets relative to the reference in inverse-sigmoid coordinates. For a four-coordinate normalized box,

b^{(l+1)}=\sigma\!\left(\operatorname{logit}(b^{(l)})+\Delta b^{(l)}\right).

A zero residual preserves the box. A positive horizontal residual increases the normalized center, but its effect depends on the current center. It is not an additive pixel offset. Centers and dimensions are individually bounded by the sigmoid; their derived corners can still lie beyond the image. The official decoder also has reference-detachment behavior that affects training gradients, which the scalar reference does not attempt to model.

The diagram separates the shared query feature from its output heads. Box regression answers where; query/text dot products supply evidence about which tokens describe the region. A stronger word score does not mechanically move the rectangle. Those behaviors are coupled by learned features and training, not by the demo's threshold controls.

Preserve a phrase without letting every category rewrite every other

Detection prompts often concatenate names: “red cup. blue bowl.” Treating that as one unrestricted sequence lets unrelated category names influence each other's representations through text self-attention. Pooling the entire prompt into one vector loses the token-level structure needed to distinguish attributes and phrases.

Grounding DINO instead uses sub-sentence features. An attention mask blocks connections between unrelated category-name groups while retaining connections within a phrase. Per-token representations survive. §3.4 and Figure 4 motivate the representation; the released mask construction defines its boundary behavior.

4. Keep phrase context, block unrelated names

Compare unrestricted text self-attention with sub-sentence attention for the authored prompt ‘red cup. blue bowl.’ Inspect a token to see exactly which keys it can read.

[CLS]redcup.bluebowl.[SEP]
Allowed text-to-text connections (query rows, key columns)
From / to[CLS]redcup.bluebowl.[SEP]
[CLS]allowblockblockblockblockblockblockblock
redblockallowallowallowblockblockblockblock
cupblockallowallowallowblockblockblockblock
.blockallowallowallowblockblockblockblock
blueblockblockblockblockallowallowallowblock
bowlblockblockblockblockallowallowallowblock
.blockblockblockblockallowallowallowblock
[SEP]blockblockblockblockblockblockblockallow

Token cup may read: red, cup, .. These blocks constrain text self-attention; image-mediated interactions can still couple concepts later.

Explicit toy tokenization, not BERT output. Each period belongs to the preceding phrase; CLS and SEP are singleton groups. Position IDs also reset within phrases upstream; this mask-only figure does not implement the tokenizer, padding or position embeddings.

Inspect “cup” under the two modes. In the grouped mask it can read “red”, “cup” and its closing period; it cannot directly read “blue” or “bowl”. The special boundary tokens have singleton blocks in this unpadded example. Upstream also resets position IDs inside each block. The figure intentionally shows the mask graph rather than pretending its hand-written token list is the output of BERT's tokenizer.

A phrase boundary is not a universal semantic parser. Real words can split into multiple subword tokens, punctuation affects grouping, and long prompts face the configured token limit. A referring expression should preserve the context needed to resolve its target rather than arbitrarily split an attribute or relationship away from its noun. The official repository recommends periods between category names.

Blocking text self-attention also does not prove complete independence between categories. Image features provide a later route for information to mix. The mask is a targeted restriction on one interaction graph, not a guarantee that adding a category leaves all other scores unchanged.

Alignment is a token prediction problem, not a fixed class vocabulary

The final query representation is compared with enhanced text tokens. In the released ContrastiveEmbed, these are dot products with padding masked out; there is no class softmax or mandatory cosine normalization in that head. Training uses focal loss over token logits, with box L1 and GIoU terms. Classification and box costs first establish bipartite matching; corresponding loss components then supervise the matched predictions. Auxiliary supervision applies to encoder outputs and decoder layers. These details appear in §3.5 and the pinned contrastive head.

This distinction explains why “contrastive” should not automatically be translated as CLIP's batchwise InfoNCE objective. Here, token-level supervision and detection matching are central. Box-to-phrase alignment requires annotation-derived token correspondences; a correct region and a plausible word in isolation are not enough.

5. Keeping a box and naming it are separate decisions

Adjust the two strict thresholds independently. A retained query can have no decoded label, and low text thresholds can join words from unrelated phrases.

0.5
0.5

Wide diagram: swipe, or focus here and use the left and right arrow keys.

Authored scene: cup on left, bowl on right, ambiguous background candidateQ0: 0.917 keptQ1: 0.881 keptQ2: 0.500 filtered
Independent sigmoid scores over content tokens
From / toredcupbluebowl
Q0: cup0.7500.9170.3100.198
Q1: bowl0.1980.3100.8180.881
Q2: background0.4500.5000.4010.354
  • Q0: retained; label tokens: red cup.
  • Q1: retained; label tokens: blue bowl.
  • Q2: filtered; label tokens: (no output).

Scores do not sum to one across words. At an exact score/threshold tie, strict “greater than” excludes the candidate. Q2 has a maximum score of exactly 0.5.

Authored token logits and scene, not model detections. This illustrates the default demo’s independent thresholds with explicit special-token exclusion. COCO evaluation uses a different phrase/category aggregation path; these controls are not an AP evaluator.

The default demo decoding path applies sigmoid, retains a query when its maximum token score exceeds box_threshold, and decodes tokens above text_threshold. The thresholds answer different questions. Raise only the text threshold above every token score and a retained box can have an empty decoded phrase. Lower it and tokens from multiple category groups can appear together. The optional remove_combined path changes phrase decoding, so it should not be silently conflated with the default.

The explorer explicitly excludes boundary/punctuation tokens when constructing labels. That teaching policy is separate from upstream padding masking and token-decoder handling. It also uses authored logits: changing thresholds cannot repair an underlying failure to recognize a red attribute, distinguish two cups, or resolve a spatial relationship.

Read results as protocols, not as a single score

“Open-set” describes the ability to specify targets through language. It does not establish that every evaluated concept was absent from every pretraining source. The conference paper defines zero-shot in terms of not using the training split of the evaluation dataset, as clarified in the ECCV publication abstract footnote. Objects365 can cover concepts also present in COCO. Dataset transfer and semantic novelty are different claims.

6. A result belongs to a training-to-evaluation path

COCO zero-shot transfer

  1. 1. Training dataO365 + OI + GoldG → Swin-L
  2. 2. EvaluateCOCO 2017val
  3. 3. Cited result52.5 box AP

No COCO training split used for this row. Categories may already occur in upstream data; zero-shot here is dataset transfer, not proof of unseen semantic classes.

Published measurements from paper v5, Tables 2 and 4. No local inference, timing, or metric computation.

The 52.5 AP COCO result belongs to the Swin-L model pretrained on O365, OpenImages and GoldG. In Table 2, the Swin-T model with O365, GoldG and Cap4M instead obtains 48.4 zero-shot AP and 57.2 after COCO fine-tuning. The official checkpoint table identifies that smaller released model. Downloading it does not reproduce the headline large-model result.

There is also a reporting wrinkle worth preserving: the same README's COCO evaluation instructions say the example should obtain 48.5, while its checkpoint table says 48.4. This review reports the paper/table value and retains the discrepancy rather than claiming a new local measurement. Resolve exact transforms, checkpoint bytes and evaluation code before demanding a one-decimal reproduction.

Even within Swin-T, comparing 46.7 AP with O365 to 48.1 AP with O365+GoldG changes the training data. It supports the usefulness of grounding data in that reported setting; it does not isolate an architectural change. Likewise, comparing Swin-T and Swin-L changes both capacity and, in these rows, data. Absolute scores alone cannot establish which fusion mechanism caused a gain.

ODinW broadens the question to domain transfer. Table 4 reports a Swin-L mean of 26.1 and median of 18.4 across downstream datasets with a mixture that includes COCO and RefC. Those numbers are neither COCO AP nor pooled AP across all boxes. A mean can hide poorly served domains; the median is useful additional context, and per-dataset results matter for an intended deployment.

Long-tail coverage remains uneven. Table 3 reports 25.6 LVIS minival AP for Grounding DINO T with O365+GoldG versus 24.9 for GLIP-T(C), but rare-category AP is 14.4 versus 17.7. Higher overall AP does not imply uniformly stronger open-vocabulary recognition. The same table includes different data mixtures and fine-tuned rows, which should be separated before making a ranking claim.

Recognizing a name is easier than resolving an expression

Referring expression comprehension asks for the object specified by a phrase, not every instance of a named class. Grounding DINO uses its highest-scoring output for this setting. A phrase such as “the cup behind the bowl” tests relational grounding that a category-only prompt never exercises. The metric in Table 5 is top-1 accuracy, not detection AP.

The paper explicitly finds limited referring performance without REC data and much larger gains when RefCOCO/+/g supervision is added. Swin-T with O365+GoldG reports 50.41 on RefCOCO val; the row with RefC added reports 73.98, and the additionally fine-tuned row reports 89.19. They are different training protocols, not three interchangeable measures of zero-shot language understanding.

The table also flags possible image overlap for its strongest COCO-inclusive large-model row because COCO can contain RefC validation images. Different annotations do not make identical images disjoint. Any stronger claim about clean generalization must inspect image identity, split construction and training sources, not merely a dataset label. This is particularly relevant when using such a detector to create pseudo-labels for a new evaluation corpus.

A reference you can run without downloading a detector

The implementation below keeps five mechanisms small enough to test exactly: two attention normalization axes, max-then-top-K selection, sub-sentence mask connectivity, logit-space box refinement and independent token thresholds. It uses Python's standard library and deterministic authored inputs. It provides no checkpoint, tokenizer, image backbone, training loop, matching solver or benchmark evaluator.

Run python3 examples/grounding-dino.py from the repository root. The JSON fixture is also compared against the browser helpers by node --test test/grounding-dino.test.mjs. Setup and limits are documented in examples/README-grounding-dino.md; examples/requirements-groundingDino.txt intentionally contains no packages. The panels are generated from named regions in the same canonical Python file, so they remain executable source rather than separately maintained pseudocode.

These five panels are extracted from examples/grounding-dino.py, a deterministic standard-library reference. The inputs are authored features and logits. The code isolates arithmetic; it does not encode an image, train a detector, or reproduce paper metrics. Official implementation links are pinned to 856dde20.

1. Normalize the two reading directions independently

Original paper

§3.1 and Figure 3 · feature enhancement · p. 5, Fig. 3 block 2 · feature enhancer

The feature enhancer combines within-modality processing and bidirectional cross-modal fusion. The released fusion block shares compatibility scores between its two reading directions.

Math and tensor shapes

\begin{aligned}A&=IT^\top/\sqrt{d_h}\\U_I&=\operatorname{softmax}_{t}(A)T\\U_T&=\operatorname{softmax}_{i}(A^\top)I\end{aligned}

I [6,2], T [2,2] → A [6,2]; updates [6,2] and [2,2]

One head with identity projections makes the axis choice inspectable. Transposing row-normalized weights would be wrong: the reverse direction normalizes over spatial keys, not words. This omits learned query/key/value/output projections, stabilization clamps, dropout, residual layer scales, normalization, and the rest of the enhancer. The pooled values are not final enhanced features.

Pinned official source

Python implementation

1. Normalize the two reading directions independently
python
def bidirectional_attention(image, text):
    """Pool supplied projected values in both directions from shared scores."""
    check_matrix(image)
    check_matrix(text)
    if len(image[0]) != len(text[0]):
        raise ValueError("Channel dimensions must agree")
    scale = math.sqrt(len(image[0]))
    scores = [[sum(a * b for a, b in zip(v, t)) / scale
               for t in text] for v in image]
    image_to_text = [softmax(row) for row in scores]
    text_to_image = [softmax(list(column)) for column in zip(*scores)]

    def pool(weights, values):
        return [[sum(w * v[d] for w, v in zip(row, values))
                 for d in range(len(values[0]))] for row in weights]

    return {"scores": scores, "imageToText": image_to_text,
            "textToImage": text_to_image,
            "imageUpdate": pool(image_to_text, text),
            "textUpdate": pool(text_to_image, image)}

2. Reduce text before ranking spatial positions

Original paper

§3.2, Eq. 1 · language-guided query selection · p. 7, Eq. 1 · language-guided query selection

Image-token and text-token dot products determine which encoder positions initialize positional queries. Query content remains learned.

Math and tensor shapes

\begin{aligned}s_i&=\max_{t\in V}(I_i^\top T_t)\\J&=\operatorname{TopK}_i(s_i)\end{aligned}

token logits [N_image,N_text], valid mask [N_text] → K image IDs

The input is an alignment-logit matrix, not the softmax weights from row 1. With rows [2,0], [1.4,0.2], [0.1,1.8] and both tokens valid, K=2 selects spatial IDs [0,2]. Excluding the second text token instead selects [0,1]. There is no balancing per phrase or overlap suppression. The helper caps K at the number of positions and breaks ties by index, both explicit reference conventions.

Pinned official source

Python implementation

2. Reduce text before ranking spatial positions
python
def select_queries(scores, valid_tokens, k):
    """Rank image positions by their maximum valid-token logit."""
    check_matrix(scores)
    if len(valid_tokens) != len(scores[0]) or not any(valid_tokens):
        raise ValueError("At least one valid text token required")
    if not isinstance(k, int) or k < 0:
        raise ValueError("K must be a nonnegative integer")
    ranked = [{"index": i, "score": max(
        x for x, valid in zip(row, valid_tokens) if valid)}
              for i, row in enumerate(scores)]
    return sorted(ranked, key=lambda item: (-item["score"], item["index"]))[:k]

3. Specify an allowed-attention graph for text

Original paper

§3.4 and Figure 4 · sub-sentence features · p. 6, Fig. 4 · sentence, word and sub-sentence features

Token features retain local phrase context while category boundaries prevent unrelated names from attending directly to one another.

Math and tensor shapes

\begin{aligned}M_{ab}&=\mathbf{1}[g_a=g_b]\\A_{ab}&=\begin{cases}\ell_{ab}&M_{ab}=1\\-\infty&M_{ab}=0\end{cases}\end{aligned}

group IDs [8] → allowed attention [8,8]

For [CLS], red, cup, period, blue, bowl, period, [SEP], use group IDs [0,1,1,1,2,2,2,3]. A period closes and belongs to its preceding block. Special tokens are singleton groups. This tiny function only builds the mask from supplied groups; upstream also constructs token-to-category maps and resets position IDs. Padding masks and special-token output filtering are separate contracts.

Pinned official source

Python implementation

3. Specify an allowed-attention graph for text
python
def phrase_mask(groups):
    """Construct allowed attention from explicit phrase-group IDs.

    CLS and SEP have unique groups; each period belongs to its preceding
    phrase. This describes one unpadded prompt, not BERT tokenization.
    """
    return [[a == b for b in groups] for a in groups]

4. Refine normalized boxes in logit coordinates

Original paper

§3.3 · cross-modality decoder and box prediction · p. 5, Fig. 3 · decoder architecture (box update is code detail)

Figure 3 locates the decoder and box outputs; it does not specify the logit-coordinate update. In the released code, each box head predicts a residual relative to a four-coordinate reference box after decoder feature updates.

Math and tensor shapes

b^{(l+1)}=\sigma\!\left(\operatorname{logit}(b^{(l)})+\Delta b^{(l)}\right)

reference [cx,cy,w,h], residual [4] → refined normalized box [4]

Zero residual is the identity. A positive horizontal residual increases cx without altering cy, w or h; it does not add that many pixels. Each coordinate lies in (0,1) mathematically, but box corners need not remain inside the image. The stdlib helper rejects endpoint references, whereas upstream inverse_sigmoid clips them. Forward arithmetic does not model detached reference feedback, gradients, multi-layer supervision or learned deltas.

Pinned official source

Python implementation

4. Refine normalized boxes in logit coordinates
python
def refine_box(reference, delta):
    """Apply four residuals in logit space to (cx, cy, w, h)."""
    if (len(reference) != 4 or len(delta) != 4
            or not all(math.isfinite(x) and 0 < x < 1 for x in reference)
            or not all(math.isfinite(x) for x in delta)):
        raise ValueError("Expected interior 4D reference and finite deltas")
    return [sigmoid(math.log(x / (1 - x)) + change)
            for x, change in zip(reference, delta)]

5. Decode independent token scores without a class softmax

Original paper

§3.5 · token alignment; released inference.py · demo decoding · p. 8, §3.5 · token alignment (demo thresholds are code detail)

The training head dots object queries with text tokens. Focal classification, box L1 and GIoU costs participate in matching and supervision. The released demo thresholds sigmoid scores.

Math and tensor shapes

\begin{aligned}p_{qt}&=\sigma(q_q^\top t_t)\\\mathrm{keep}_q&=[\max_t p_{qt}>\tau_b]\\L_q&=\{t:p_{qt}>\tau_t\}\end{aligned}

logits [queries,tokens] → retained flags, max scores, label-token IDs

A zero logit is exactly 0.5, and strict equality at threshold 0.5 is excluded. Box retention and label construction use independent thresholds; a kept query can have an empty label. The explicit valid-token mask excludes special tokens in this teaching example. Upstream ContrastiveEmbed masks padding, and phrase extraction has its own special-token handling. The default demo may combine multiple phrases; remove_combined offers a different decoding policy. This helper does not compute a training loss, Hungarian matching or COCO AP.

Pinned official source

Python implementation

5. Decode independent token scores without a class softmax
python
def decode_alignment(logits, box_threshold, text_threshold, valid_tokens):
    """Apply independent strict box and label thresholds to token scores."""
    check_matrix(logits)
    if not all(math.isfinite(t) and 0 <= t <= 1
               for t in (box_threshold, text_threshold)):
        raise ValueError("Thresholds must lie in [0, 1]")
    if len(valid_tokens) != len(logits[0]) or not any(valid_tokens):
        raise ValueError("Valid tokens required")
    result = []
    for i, row in enumerate(logits):
        scores = [sigmoid(x) for x in row]
        score = max(s for s, valid in zip(scores, valid_tokens) if valid)
        result.append({"index": i, "score": score,
                       "kept": score > box_threshold,
                       "tokens": [t for t, s in enumerate(scores)
                                  if valid_tokens[t] and s > text_threshold]})
    return result

Original paper excerpts: Shilong Liu et al., Grounding DINO (arXiv:2303.05499v5). Cropped/rasterized from the original PDF. Source PDF · arXiv distribution license

What the architecture gives you, and what remains to verify

The main contribution is a coherent path for language to affect detection: enhance features together, initialize positions using token compatibility, and keep both modalities available during refinement. This is more substantial than replacing class names after a closed-set prediction. At the same time, open-ended input text does not guarantee calibrated confidence, exhaustive object coverage, robust counting, or relational understanding on a new domain.

For an application, hold the checkpoint and resize policy fixed, record the exact prompt and tokenizer behavior, and evaluate box quality and phrase correctness separately. Vary prompt wording and category-list composition because early fusion and token ranking can change the candidate set. Preserve raw scores when assessing thresholds. If a dataset contains several cameras or repeated scenes, split by their identities before generating or evaluating pseudo-labels; nearby frames are not independent evidence of generalization.

Grounding DINO predicts boxes and alignments. Segmentation systems built by handing its boxes to SAM add a distinct model and evaluation boundary. This review makes no claims about those downstream masks, and none of the browser diagrams constitutes measured detector behavior.

  • Deformable DETR explains sparse multi-scale image reads and reference coordinates behind the image attention used here.
  • RF-DETR offers a different detector-design perspective centered on architecture and deployment tradeoffs.
  • Modern object detection places set prediction alongside proposal and dense detection approaches.
  • Cross-attention develops the query/key/value roles behind the two reading directions.
  • Contrastive loss provides background for alignment objectives; Grounding DINO's focal token objective needs the distinction explained above.
  • Official Grounding DINO code at revision 856dde20 and paper v5 are the primary implementation and scientific references used throughout this review.

If you found this paper review helpful, consider sharing it with others.

Mastodon