Segment Anything Model (SAM)

TL;DR

SAM frames image segmentation as a promptable task: given an image and a geometric prompt (point, box, or mask), produce a valid segmentation mask. The paper also explores text prompting with a separate experimental pathway; the released original checkpoint does not expose a text encoder. A ViT-H image encoder computes the image embedding once, then a lightweight mask decoder produces masks in real time (~50ms) for any prompt. Trained on SA-1B — 1.1 billion masks across 11 million images, built via a three-stage data engine — SAM achieves strong zero-shot transfer to unseen tasks and distributions without fine-tuning. The contribution is not a single architectural novelty but the combination of task definition, data engine, and scale that makes segmentation a foundation-model problem.

The Foundation Model Framing

SAM applies the NLP foundation model playbook to segmentation. In NLP, GPT and BERT defined broad pretraining tasks (next-token prediction, masked language modeling) that transfer to diverse downstream tasks via prompting. SAM does the same for segmentation: define a single task general enough to serve as pretraining, train at massive scale, then transfer via prompt engineering at inference.

The key question is: what is the right "pre-trainable" task for segmentation? The authors propose promptable segmentation — given any segmentation prompt, return a valid mask. This is deliberately underspecified: the prompt may be ambiguous (a point on an object could mean the part or the whole), so the model must handle ambiguity gracefully rather than forcing a single interpretation.

This framing has a subtle but important consequence for the training objective. Traditional segmentation models optimize for a fixed label set on a fixed dataset. SAM instead optimizes for prompt-conditional mask prediction, which means the model must learn a general correspondence between spatial prompts and object boundaries rather than memorizing category-specific segmentation patterns.

The Promptable Segmentation Task

Formally, the task is a function f: (\mathbf{I}, \mathbf{p}) \rightarrow \mathbf{M} that maps an image \mathbf{I} and a prompt \mathbf{p} to a set of valid segmentation masks \mathbf{M}. "Valid" means any mask that a reasonable annotator would produce given the same prompt — the model is not required to guess the user's intent when the prompt is ambiguous.

This task subsumes several existing segmentation tasks. Interactive segmentation (clicks to masks), edge detection (dense points to boundaries), object proposal generation (grid of prompts to candidate masks), and instance segmentation (box prompts to masks) are all special cases of promptable segmentation with different prompt types. This generality is what makes it suitable as a pretraining task — a model that solves promptable segmentation well has implicitly learned the sub-skills needed for all these downstream tasks.

Architecture: Three Components

SAM decomposes into three modules with an asymmetric compute design: a heavy image encoder that runs once per image, and lightweight prompt encoder + mask decoder that run per prompt.

Image Encoder (ViT-H). A Vision Transformer pretrained with MAE (Masked Autoencoder), specifically ViT-Huge (632M parameters, 32 transformer blocks, embedding dimension 1280). The architecture uses 14×14 windowed attention in most blocks with four interleaved global attention blocks to capture long-range dependencies. The input image is resized with its longest side at 1024 pixels, preserving aspect ratio, then padded to 1024×1024 and the encoder produces a 64 \times 64 \times 256 feature map (after a neck that reduces the channel dimension from 1280 to 256). This is the expensive step (~0.15s on an A100), but it only runs once per image. All subsequent prompt interactions reuse this embedding.

Prompt Encoder. Handles two categories of prompts:

• Sparse prompts (points and boxes in the released model): mapped to 256-d vectors using fixed random Fourier positional features plus learned type embeddings. Points have distinct foreground/background types; boxes have distinct top-left and bottom-right corner types. The paper’s separate text experiments use a CLIP text encoder, which is not included in the released geometric prompt encoder.
• Dense prompts (masks): supplied at 256 \times 256 and mapped to a 256-d spatial embedding via two 2 \times 2 convolutions with stride 2, producing a 64 \times 64 \times 256 feature map that is element-wise summed with the image embedding.

Mask Decoder. A modified transformer decoder with only two blocks — deliberately kept small so that the prompt-to-mask step is fast. Each block performs four operations: (1) self-attention on prompt tokens, (2) cross-attention from prompt tokens to the image embedding, (3) point-wise MLP on each token, (4) cross-attention from the image embedding to prompt tokens. The bidirectional cross-attention (steps 2 and 4) is the key design choice: it lets the image features attend to prompt information and vice versa, unlike a standard decoder that only attends in one direction.

After the two blocks, the image embedding is upsampled 4\times via two transposed convolution layers (each 2\times) and each output token produces a mask prediction as per-pixel logits via a spatial dot product with the upsampled features. The native API resizes and crops those logits before thresholding at zero for binary masks. The lightweight decoder enables interactive use: the paper reports approximately 50ms per prompt in a web browser once the image embedding is available.

Ambiguity-Aware Output

A single point prompt is inherently ambiguous — it could refer to a subpart, the whole object, or the entire scene. SAM handles this by predicting three alternative masks simultaneously, each with a predicted IoU score. Whole, part, and subpart are common nested interpretations, not fixed labels assigned to the three heads. For a deeper exploration of how this multi-mask strategy resolves ambiguity in practice, see the SAM multi-mask ambiguity article. During training, only the mask with the lowest loss against the ground truth receives gradients:

The mask objective combines focal loss and Dice loss in a 20:1 ratio. The ambiguity-aware mask objective backpropagates the lowest mask loss rather than averaging incompatible interpretations. A separate IoU prediction head is supervised with mean-squared error against the predicted mask’s measured ground-truth overlap, with loss weight 1.0 (Appendix A). At inference, the native multimask API returns all three alternatives and their predicted IoUs; selecting the highest score is an application policy. The dedicated single-mask mode uses an additional fourth mask token. These scores are ranking regressions, not calibrated probabilities.

Training simulates an initial foreground point or noisy ground-truth box, then eight corrective point interactions sampled from the difference between the predicted mask and ground truth. False-negative regions yield foreground points; false-positive regions yield background points. The previous prediction’s unthresholded low-resolution mask logits also become a dense mask prompt. Two additional mask-only interactions bring the total to 11 rounds, teaching SAM to refine a prediction without always requiring another click (Appendix A).

Implementation: From Image Pixels to Prompted Masks

The architecture and ambiguity policy now form a concrete inference pipeline. The reference below follows the original released SAM modules, showing where image geometry is cached, how each prompt enters the decoder, and how logits return to original pixels. The small offline profile checks the mechanics; the real-photo artifacts use the official pretrained ViT-B model. The browser demos above illustrate the ideas, while these saved masks come from actual native inference.

This executed PyTorch reference uses verified, unchanged original SAM modeling modules. Offline smoke runs a small native random-weight architecture to test mechanics. The saved examples below use the authenticated official ViT-B checkpoint on real CC0 photos. Neither profile reproduces SA-1B training or the paper’s benchmarks.

Python reference · Native source and Apache-2.0 license · Pinned requirements · Real-photo exporter · Reproduction instructions

unzip sam_native.zip
python sam.py --smoke
python sam_export.py --checkpoint sam_vit_b_01ec64.pth --output sam-demo

Place the extracted sam_native directory beside sam.py. Install the pinned requirements in your own environment; CPU wheels suffice. Smoke requires no network or weights. The exporter verifies the full checkpoint SHA256 before loading every state tensor strictly and can fetch the pinned photos. It imports only native modeling/build modules, bypassing optional automatic-mask dependencies. Released SAM accepts points, boxes, and masks; the paper’s text experiments are separate.

1. Keep geometry intact while forming the square input

Original paper

Enlarge figure

Math and tensor shapes

s=1024/\max(H,W)

x_c=(I_c-\mu_c)/\sigma_c

Pillow bilinear resizing matches the native NumPy/PIL prediction path. Keep pixels in 0..255, subtract the RGB mean, divide by the RGB standard deviation, then append zeros at the right and bottom. Padding after normalization gives zero-valued model input, rather than a normalized black border. Keep the actual rounded dimensions for transforming prompts and removing padding.

RGB [3,H,W] → [1,3,1024,1024]; coffee: 400×600 → 683×1024

Exact native operations

Python implementation

python

def preprocess(model, image):
    """PIL bilinear resize, RGB 0..255 normalization, then right/bottom padding."""
    image = image.convert("RGB")
    original_size = (image.height, image.width)
    target = model.image_encoder.img_size
    input_size = resized_shape(original_size, target)
    resized = image.resize(input_size[::-1], Image.Resampling.BILINEAR)
    pixels = torch.from_numpy(np.array(resized)).permute(2, 0, 1).float()
    normalized = (pixels - model.pixel_mean) / model.pixel_std
    padded = F.pad(
        normalized, (0, target - input_size[1], 0, target - input_size[0])
    )
    return padded.unsqueeze(0), input_size, original_size

2. Encode the image once and reuse its spatial features

Original paper

Enlarge figure

Math and tensor shapes

E=\operatorname{ImageEncoder}(x)

ViT-B has 12 blocks, width 768, patch size 16, and a neck projecting to 256 channels. Windows and four global-attention blocks retain the native architecture. The cache stores both features and image geometry. This inference reference freezes the encoder and disables gradients; that is an inference/reuse choice, not a claim that original SAM training froze the backbone.

Official ViT-B: [1,256,64,64]; offline reduced profile: [1,32,4,4]

Exact native operations

Python implementation

python

@torch.no_grad()
def encode_image(model, image):
    """Cache once; subsequent prompts reuse the same tensor without re-encoding."""
    padded, input_size, original_size = preprocess(model, image)
    embedding = model.image_encoder(padded)
    return {
        "embedding": embedding,
        "input_size": input_size,
        "original_size": original_size,
        "padded": padded,
    }

3. Add point or corner identity to Fourier position features

Original paper

Enlarge figure

Math and tensor shapes

a=2\pi(2u-1)G

\phi(u)=[\sin(a),\cos(a)]

z=\phi(u)+e_{type}

First scale x and y with the actual resized width and height. Then shift by half a pixel and normalize in the padded square. G is a saved Gaussian frequency buffer, not a learned coordinate table. Label 1 is foreground, 0 background, and −1 is a padding token whose position features are discarded. Point-only requests append that padding token; boxes instead contribute two distinct learned corner types. Native Fourier code is linked below.

One point → [B,2,256] with padding; one point + box → [B,3,256]

Exact native operations · Fourier position encoding

Python implementation

python

def encode_sparse(encoder, points, labels, boxes=None):
    """Match native Fourier position plus point/corner type embeddings."""
    if (
        points.ndim != 3
        or points.shape[-1] != 2
        or labels.shape != points.shape[:2]
    ):
        raise ValueError("Expected points [B,N,2] and labels [B,N]")
    if not torch.isin(labels, labels.new_tensor([-1, 0, 1])).all():
        raise ValueError(
            "Point labels must be -1 padding, 0 background, or 1 foreground"
        )
    if not torch.isfinite(points).all():
        raise ValueError("Point coordinates must be finite")
    if boxes is None:
        points = torch.cat(
            [points, points.new_zeros(points.shape[0], 1, 2)], dim=1
        )
        labels = torch.cat(
            [labels, labels.new_full((labels.shape[0], 1), -1)], dim=1
        )
    sparse = encoder.pe_layer.forward_with_coords(
        points + 0.5, encoder.input_image_size
    )
    sparse[labels == -1] = encoder.not_a_point_embed.weight
    sparse[labels == 0] += encoder.point_embeddings[0].weight
    sparse[labels == 1] += encoder.point_embeddings[1].weight
    if boxes is not None:
        if (
            boxes.shape != (points.shape[0], 4)
            or not torch.isfinite(boxes).all()
        ):
            raise ValueError("Expected finite boxes [B,4]")
        if (boxes[:, 2:] < boxes[:, :2]).any():
            raise ValueError("Box maximum must follow its minimum")
        corners = encoder.pe_layer.forward_with_coords(
            boxes.reshape(-1, 2, 2) + 0.5, encoder.input_image_size
        )
        corners[:, 0] += encoder.point_embeddings[2].weight
        corners[:, 1] += encoder.point_embeddings[3].weight
        sparse = torch.cat([sparse, corners], dim=1)
    return sparse

4. Supply a spatial mask prompt or the no-mask vector

Original paper

Enlarge figure

Math and tensor shapes

D=\operatorname{MaskEncoder}(m)

E_0=E+D

Feed the previous decoder’s unthresholded 256-square logits back for refinement. The native mask encoder reduces resolution by four while expanding channels 1→4→16→256. If no mask is supplied, broadcast one learned vector over every spatial location. This dense input is added to image features before the transformer, unlike the sparse tokens concatenated to its token sequence.

[B,1,256,256] → [B,256,64,64]; no mask broadcasts [1,256,1,1]

Exact native operations

Python implementation

python

def encode_dense(encoder, batch_size, mask_logits=None):
    """Use low-resolution logits for refinement or the learned no-mask vector."""
    if batch_size <= 0:
        raise ValueError("Prompt batch must be positive")
    if mask_logits is not None:
        expected = (batch_size, 1, *encoder.mask_input_size)
        if (
            mask_logits.shape != expected
            or not torch.isfinite(mask_logits).all()
        ):
            raise ValueError(f"Expected finite mask logits {expected}")
        return encoder.mask_downscaling(mask_logits)
    return encoder.no_mask_embed.weight.reshape(1, -1, 1, 1).expand(
        batch_size, -1, *encoder.image_embedding_size
    )

5. Let tokens and image positions update each other

Original paper

Enlarge figure

Math and tensor shapes

T\leftarrow\operatorname{Attn}(T,E,E)

E\leftarrow\operatorname{Attn}(E,T,T)

A=\operatorname{softmax}(QK^\top/d_h^{1/2})

The equations name attention direction, omitting residuals, projections, and normalization for clarity. Native blocks retain those details, including the first block’s special self-attention path and positional features re-added at attention. Eight heads use reduced 128-channel cross-attention projections. An image embedding is repeated across prompt requests, then each copy receives its own dense prompt. The linked native block shows both directions explicitly.

1 IoU + 4 mask + 2 point/padding tokens: [B,7,256]; image [B,4096,256]

Exact native operations

Python implementation

python

def two_way(decoder, embedding, position, sparse, dense):
    """Expose native token setup and its two-way transformer without changing it."""
    if embedding.shape[0] != 1:
        raise ValueError("Reuse one image embedding per prompt batch")
    output = torch.cat([decoder.iou_token.weight, decoder.mask_tokens.weight])
    output = output.unsqueeze(0).expand(sparse.shape[0], -1, -1)
    tokens = torch.cat([output, sparse], dim=1)
    source = embedding.repeat_interleave(tokens.shape[0], dim=0) + dense
    position = position.repeat_interleave(tokens.shape[0], dim=0)
    # Each native layer: self-attn, token->image, token MLP, image->token.
    # Then native final token->image attention and LayerNorm.
    updated_tokens, updated_image = decoder.transformer(
        source, position, tokens
    )
    return updated_tokens, updated_image, source.shape

6. Turn output tokens into four spatial mask logits

Original paper

Enlarge figure

Math and tensor shapes

v_i=\operatorname{MLP}_i(t_i)

\ell_i(x,y)=v_i^\top U(x,y)

q=\operatorname{IoUHead}(t_{IoU})

Upsampling changes channels 256→64→32 while doubling each spatial axis twice. A three-layer MLP maps each 256-dimensional mask token to a 32-dimensional vector. Multiplying those vectors by flattened upsampled features produces per-pixel logits. There is no sigmoid in the native returned logits. The additional token passes through another MLP to regress one predicted IoU per mask.

U [B,32,256,256]; vectors [B,4,32]; logits [B,4,256,256]; q [B,4]

Exact native operations

Python implementation

python

def generate_masks(decoder, tokens, image_tokens, image_shape):
    """Hypernetwork vectors dot upsampled features; the IoU token regresses quality."""
    batch, channels, height, width = image_shape
    spatial = image_tokens.transpose(1, 2).reshape(
        batch, channels, height, width
    )
    upscaled = decoder.output_upscaling(spatial)
    mask_tokens = tokens[:, 1 : 1 + decoder.num_mask_tokens]
    vectors = torch.stack(
        [
            mlp(mask_tokens[:, i])
            for i, mlp in enumerate(decoder.output_hypernetworks_mlps)
        ],
        dim=1,
    )
    logits = (vectors @ upscaled.flatten(2)).reshape(
        batch, decoder.num_mask_tokens, *upscaled.shape[-2:]
    )
    scores = decoder.iou_prediction_head(tokens[:, 0])
    return logits, scores

7. Preserve ambiguity, then make ranking an explicit policy

Original paper

Enlarge figure

Math and tensor shapes

i^*=\arg\max_i q_i

The three alternatives are not fixed semantic labels such as head, animal, or scene. Native multimask output returns all three with their scores. This reference additionally computes argmax as an application policy; single-mask mode uses token zero rather than the best of the three. Predicted IoU is regression used for ranking, not a calibrated probability or measured ground-truth overlap. No clipping or softmax is applied.

multimask: tokens 1..3 → [B,3,256,256]; single: token 0 → [B,1,256,256]

Exact native operations

Python implementation

python

def select_masks(logits, scores, multimask=True):
    """Keep the dedicated single token or all three ambiguity tokens, then rank."""
    if (
        logits.ndim != 4
        or logits.shape[1] != 4
        or scores.shape != logits.shape[:2]
    ):
        raise ValueError(
            "Expected four native masks and matching predicted IoUs"
        )
    if not torch.isfinite(logits).all() or not torch.isfinite(scores).all():
        raise ValueError("Mask logits and quality predictions must be finite")
    selected = slice(1, 4) if multimask else slice(0, 1)
    logits, scores = logits[:, selected], scores[:, selected]
    # Explicit application policy; native multimask API does not pick a winner.
    best = scores.argmax(dim=1)
    return logits, scores, best

8. Remove padding before returning masks to original pixels

Original paper

Enlarge figure

Math and tensor shapes

L_0=\operatorname{Resize}_{square}(\ell)

L_1=\operatorname{Crop}(L_0)

L=\operatorname{Resize}_{original}(L_1)

M(x,y)=[L(x,y)>0]

Both interpolations are bilinear with align_corners=False. Cropping before the final resize keeps the padded border from being stretched into the image. Thresholding only at the end retains smooth logit interpolation and matches the native zero logit threshold. The tests check rectangular images and a padded region with deliberately extreme logits to catch the wrong crop order.

[B,3,256,256] → square 1024 → resized H′×W′ → original H×W

Exact native operations

Python implementation

python

def postprocess(logits, target, input_size, original_size):
    """Upscale logits to padded square, crop padding, resize back, then threshold."""
    if logits.ndim != 4 or min(logits.shape) <= 0:
        raise ValueError("Expected nonempty logits [B,K,H,W]")
    if (
        min(input_size) <= 0
        or max(input_size) > target
        or min(original_size) <= 0
    ):
        raise ValueError("Invalid input or original dimensions")
    square = F.interpolate(
        logits, (target, target), mode="bilinear", align_corners=False
    )
    cropped = square[..., : input_size[0], : input_size[1]]
    full = F.interpolate(
        cropped, original_size, mode="bilinear", align_corners=False
    )
    return full, full > 0.0

Inspect genuine point-prompt outputs

Each photo was encoded once, prompted with one foreground point, and decoded into three alternatives. White dots mark prompts in the saved best-score overlays. The selector displays each actual binary mask; scores are predicted IoU regressions. A second mask-logit refinement reused each embedding. All three native/wrapper maximum errors were exactly 0.0 on both photos: low-resolution logits, scores, and full-size logits.

Chelsea the cat

Foreground point (x,y): 240, 160. Original 451×300 pixels.

Inspect a returned maskCandidate 1: predicted IoU 0.988Candidate 2: predicted IoU 0.955Candidate 3: predicted IoU 0.847

Foreground area: 104,248 pixels. Predicted IoU: 0.9884.

Photo: Stefan van der Walt, CC0. Pinned source photo · License record · Logits, masks, scores, and prompts

Cup of coffee

Foreground point (x,y): 285, 215. Original 600×400 pixels.

Inspect a returned maskCandidate 1: predicted IoU 0.836Candidate 2: predicted IoU 0.943Candidate 3: predicted IoU 0.866

Foreground area: 37,486 pixels. Predicted IoU: 0.9428.

Photo: Rachel Michetti, CC0. Pinned source photo · License record · Logits, masks, scores, and prompts

Complete provenance and numeric record · Native revision dca509fe793f601edb92606367a655c15ac00fdf. Torch 2.12.0+cpu, CPU, 2 threads. These two photos demonstrate real inference, without ground-truth IoU evaluation.

The code makes prompt-to-mask inference inspectable. The next question is how a model with this interface acquired its generality: SAM’s data engine supplies the training masks at scale.

The Data Engine: Three Stages

The SA-1B dataset was not hand-curated from scratch. Instead, the authors built a data engine — a model-in-the-loop annotation pipeline that progressively reduces human effort across three stages:

1. Assisted-Manual (Stage 1). Annotators label masks using a browser-based tool powered by an early SAM model. SAM proposes masks, annotators correct them. 120k images, 4.3M masks. Average annotation time: 34 seconds per mask.

2. Semi-Automatic (Stage 2). SAM generates confident masks automatically; annotators label only the remaining unannotated objects. This increases object diversity by focusing human effort on objects SAM missed. 180k images, 5.9M masks (both auto and manual). Annotation time dropped to 14 seconds per mask.

3. Fully Automatic (Stage 3). A 32×32 grid of point prompts (1,024 points per image) is applied. For each point, the model predicts three masks at different granularities. Masks are filtered by predicted IoU (threshold 0.88), deduplicated via NMS with an IoU threshold, and stability-filtered by checking whether the mask changes under small perturbations of the logit threshold. No human annotation. 11M images, 1.1B masks. This stage produces the bulk of SA-1B.

The final SA-1B dataset contains 1.1 billion masks on 11 million licensed images — 400× more masks than the next largest segmentation dataset (Open Images V5, 2.8M masks). The median image contains ~100 masks, with substantially better coverage of small and medium objects compared to prior datasets. Human quality ratings showed 94% of automatically generated masks had IoU > 0.90 compared to professional annotations.

Key Results

These benchmark results use the paper's main ViT-H model and arXiv PDF v1, rather than the independent local ViT-B examples above.

Zero-shot single-point segmentation. Figure 9 and Section 7.1 evaluate one foreground point across 23 datasets. SAM's top-ranked mask exceeds RITM on 16 datasets. The oracle uses ground truth to choose the best of the three predicted masks for the same single point, rather than adding prompts; it exceeds RITM on all 23 datasets. The RITM baseline is HRNet32 IT-M trained on COCO + LVIS, not a checkpoint trained separately on each test dataset (Appendix D.1). These comparisons also expose ambiguity between valid masks and a dataset's single annotated target.

Zero-shot edge detection. Table 3 reports BSDS500 ODS F1 of 0.768 for mask-derived edges without edge-specific training. This is below the supervised HED result of 0.788 and EDETR result of 0.840; the paper discusses SAM's high recall and extra unannotated edges.

Zero-shot object proposals. Table 4 reports LVIS v1 mask AR@1000: SAM 59.3 overall versus ViTDet-H 63.0; medium objects 81.6 versus 80.8; large objects 86.9 versus 87.0. SAM is higher on medium objects and slightly lower on large objects in the table. ViTDet-H is a supervised detector used as a proposal generator, while SAM transfers without LVIS training.

Zero-shot instance segmentation. Table 5 uses ViTDet-detected boxes to prompt SAM: COCO mask AP is 46.5 versus ViTDet-H 51.0, a 4.5 AP-point gap; LVIS v1 is 44.7 versus 46.6, a 1.9 AP-point gap. Only the segmentation module is zero-shot; the box detector is supervised. The human quality ratings are a separate measurement from mask AP.

Ablation highlights. Figure 13 and Section 7.6 use the 23-dataset single-center-point protocol, including a separate ground-truth oracle. Larger encoders improve results, with diminishing gains from ViT-L to ViT-H. Training on 1M images, about 10% of SA-1B, gives results comparable to the full dataset; reducing to 0.1M images causes a substantial decline. The automatically generated masks alone perform only marginally below all three data-engine stages. These ablations concern the paper's training runs, not the local two-photo export.

Critical Analysis

Strengths.

• Zero-shot generalization. SAM transfers to new domains (medical imaging, satellite imagery, underwater photos) without fine-tuning. This is a direct consequence of data scale and the promptable task formulation.
• Amortized compute via decoupled architecture. The heavy ViT-H encoder runs once; the mask decoder runs in ~50ms. This enables interactive annotation tools where a user issues dozens of prompts on the same image.
• Data engine as a contribution. The three-stage annotation pipeline is a reusable methodology. It demonstrates that model-assisted annotation can bootstrap a billion-scale dataset with quality comparable to manual labeling.

Limitations.

• No semantic labels. SAM produces class-agnostic masks. It can segment an object but cannot tell you what it is. Downstream applications must combine SAM with a classifier or use an extension like Grounded-SAM.
• Struggles with thin structures. Fine-grained boundaries (bicycle spokes, fences, hair) are systematically underrepresented in training data and difficult for the 64×64 bottleneck to resolve. Mask quality degrades on these cases.
• Requires prompts at inference. Unlike fully automatic panoptic segmentation models, SAM does not segment an image without at least one prompt. The automatic mode (grid of points + filtering) is a workaround, not a principled solution.
• SA-1B distribution gaps. Despite its size, SA-1B is heavily biased toward everyday photography. Performance drops on specialized domains (medical histology, aerial imagery) where the visual statistics differ from the training distribution.
• No temporal reasoning. SAM operates on single images. Video segmentation, where temporal coherence matters, is out of scope for this version.

SAM 2 and Follow-Up Work

SAM 2 (Ravi et al., 2024) extends SAM to video by adding a memory mechanism: a memory encoder stores per-frame features, a memory bank maintains a fixed-size set of past frame representations, and cross-attention over the memory bank enables temporal consistency. SAM 2 uses Hiera (a hierarchical ViT) as the image encoder instead of ViT-H, improving efficiency. It was trained on SA-V, a new video dataset with 35.5M masks across 50.9k videos.

Grounded-SAM combines SAM with Grounding DINO to add open-vocabulary semantic labels — Grounding DINO detects boxes from text queries, SAM segments within those boxes. This two-stage composition became the de facto open-vocabulary segmentation pipeline before end-to-end alternatives emerged.

Efficiency variants. EfficientSAM and MobileSAM distill the ViT-H encoder into smaller backbones (ViT-Tiny, ViT-Small) for edge deployment, reducing the encoder from 632M to ~5–25M parameters with modest quality loss. FastSAM replaces the entire pipeline with a single-stage YOLOv8-based model that runs 50× faster.

Quality improvements. HQ-SAM adds a high-quality output token and a global-local feature fusion module to improve boundary quality on thin structures — directly addressing one of SAM's main failure modes. Semantic-SAM extends the multi-granularity output to six levels with semantic awareness.

Impact and Legacy

SAM reframed segmentation from a task-specific supervised problem into a foundation model problem. Before SAM, segmentation models were trained per-dataset (COCO, ADE20K, Cityscapes) with task-specific heads (Mask R-CNN, DeepLab). SAM showed that a single model trained at scale could match or approach these specialists in zero-shot, collapsing the need for per-domain annotation.

The practical impact was immediate: SAM became the default backbone for annotation tools (Label Studio, Roboflow, CVAT), enabling 10–20x speedups in mask labeling. It also became a building block for composed systems — Grounded-SAM for open-vocabulary segmentation, SAM + ControlNet for image editing, SAM + depth estimation for 3D reconstruction.

The deeper lesson is that scale in data (1.1B masks) and a well-chosen task formulation (promptable segmentation) can substitute for architectural novelty. SAM's architecture is not particularly novel — ViT encoder, transformer decoder, focal + dice loss are all standard components. The contribution is in the system design: the data engine, the task definition, and the engineering to make it work at scale.

Related Reading

• SAM 2: Segment Anything in Images and Videos — extends SAM’s promptable segmentation to video with a streaming memory bank that propagates masks across frames
• Florence-2 — unifies detection, segmentation, grounding, and captioning in one prompt-driven seq2seq model, expressing masks and boxes as location tokens
• DETR: End-to-End Object Detection with Transformers — transformer-based detection architecture that influenced SAM's decoder design
• CLIP: Learning Transferable Visual Models From Natural Language Supervision — provides the text encoder used in the paper's separate text prompt experiments
• Vision Transformer (ViT) — the base architecture for SAM's image encoder
• Faster R-CNN — the region proposal paradigm that SAM's promptable approach generalizes
• DINO — self-supervised ViT pretraining, relevant to SAM's MAE-pretrained backbone