Skip to main content

SAM 2: Segment Anything in Images and Videos

How SAM 2 extends promptable segmentation from images to video with a streaming memory bank and memory attention, propagating masks across frames and recovering objects after occlusion.

TL;DR

  • SAM 2 generalizes SAM’s promptable segmentation from single images to video: a single prompt on one frame propagates the mask across all subsequent frames without additional user input.
  • A streaming memory bank stores spatial memories and separate object pointers. Memory attention conditions the current frame on selected prompted frames and recent predictions; interactive propagation can also use a future prompted frame.
  • An object-presence head supports absent-object outputs. Stored context can help a trained model recover after occlusion, but a presence score does not guarantee recovery.
  • SAM 2 unifies image and video segmentation in a single real-time model, trained on the SA-V dataset (50.9k videos, 35.5M masks), and achieves strong zero-shot video object segmentation across multiple benchmarks.

From images to video

SAM defined promptable image segmentation: given a point, box, or mask prompt, produce a valid segmentation mask for any object. The model is remarkably general, but it has no notion of time — each image is processed in isolation.

SAM 2 extends this to video. Online processing consumes available frames without future evidence. Offline interactive propagation lets the user prompt or correct any frame in a recorded clip and run forward or backward; a prompted memory can then lie in the chronological future of the current frame. For image segmentation, SAM 2 simply runs the same pipeline on a single-frame clip, so images and videos share a single unified model.

The architectural change that makes this possible is a streaming memory bank paired with a memory attention module. Instead of discarding per-frame computation after producing a mask, SAM 2 stores a compact representation of each processed frame. When processing the next frame, it retrieves these stored memories and uses cross-attention to condition the current frame’s features on the object’s history. This gives the model a persistent, temporally coherent view of the object across the clip.

The model was trained on SA-V, a new video dataset built with a similar model-in-the-loop annotation engine to SA-1B, containing 50.9k videos and 35.5 million masks — by far the largest annotated video segmentation dataset. The backbone is Hiera, a hierarchical vision transformer that is more efficient than the ViT-H used in original SAM.

Streaming memory bank

Spatial memory comes from prompted conditioning frames and a bounded recent set of nonconditioning predictions. Both store mask-conditioned spatial feature maps. A separate list of object pointers, extracted from mask-decoder output tokens, supplies compact object summaries with its own selection horizon and token splitting.

The bank is not a universal FIFO of identical entries. The original code selects temporally closest conditioning frames, keeping before/after context when available, and then chooses recent traversal slots. Prompted memories are not simply discarded when a recent unprompted window fills. The illustration below fixes one prompt and forward traversal; its circles and masks are schematics, not model predictions.

For each frame, memory attention first conditions the current image features on selected stored memories, and the mask decoder predicts a mask from those conditioned features and any prompt. The memory encoder then fuses the unconditioned image features with the predicted mask to create a spatial memory for later frames. In online mode the system can run without future frames. Offline interactive mode may attend to future prompted spatial memories, while evaluation can still restrict object pointers to the traversal past.

Memory attention

Memory attention is the mechanism that connects the current frame to its stored history. It is a form of cross-attention: the current frame’s image features act as the queries, and the selected stored memory entries (prompted/recent spatial feature maps plus object-pointer embeddings) supply the keys and values.

A KV cache is a useful comparison for reusing context, but SAM 2 does not store a language decoder’s layer-by-layer key/value cache. It encodes masks with image features, selects spatial memories and decoder-derived pointers, and constructs the attention inputs with positional information. These representations have different limits and update rules; offline corrections can change the context and invalidate derived outputs.

Conditioning on more memory entries gives the model a richer view of the object’s appearance over time, which helps when the object changes scale, pose, or illumination across frames. In practice spatial memories and pointers follow distinct selection policies. Bounded attention inputs balance temporal context against compute; stored video frames, cached image features and output dictionaries have their own memory costs.

Implementation

The memory-mechanics reference below makes selection, absence and correction state explicit before returning to the trained model’s occlusion behavior. It uses supplied mask and pointer fixtures, so it cannot demonstrate segmentation or reappearance recovery.

Seeded offline CPU memory mechanics with synthetic image embeddings, supplied mask logits and supplied decoder tokens. Widths 4/8 and grids 2×2/4×4 are reduced; one cross-attention residual and three convolutions replace the trained modules. There is no Hiera, mask decoder, checkpoint, pretrained tracking, video segmentation, occlusion-recovery proof or benchmark reproduction. All equations are authored notation for the cited operations. The code follows the pinned August 2024 original SAM 2 release; paper v2 reports SAM 2.1 results and is not its code specification.

1. Separate image features from derived outputs

Original paper

Paper excerpt · PDF p. 4Official inference state · paper §4, Figure 3

The predictor caches unconditioned image features separately from conditioning and nonconditioning frame outputs. The latter include encoded memory and decoder pointers, not just image features.

Math and tensor shapes

I_t\in\mathbb R^{1\times4\times2\times2}
\mathcal C=\{t:R_t\}_{\rm prompted}
\mathcal U=\{t:R_t\}_{\rm propagated}

I_t is a cached synthetic image embedding. C and U are separate dictionaries of FrameRecord values R_t. A prompted output supersedes the derived output at the same index. The fixture retains every supplied image embedding for correction tests; the pinned predictor uses a small cache and can recompute evicted embeddings. These inputs are fixtures, not Hiera outputs from a real video.

Record: spatial [1,4,2,2], pointer [1,8], mask logits [1,1,4,4]

Python implementation

1. Separate image features from derived outputs
python
@dataclass
class FrameRecord:
    """Keep supplied masks, encoded spatial memory and decoder pointers apart."""

    spatial: torch.Tensor  # [1,4,2,2]
    pointer: torch.Tensor  # [1,8], split into two width-4 attention tokens
    mask_logits: torch.Tensor  # [1,1,4,4], absence is -1024
    conditioned: torch.Tensor  # [1,4,2,2], derived from current bank

    def __post_init__(self):
        check_tensor(self.spatial, (1, 4, 2, 2), "spatial")
        check_tensor(self.pointer, (1, 8), "pointer")
        check_tensor(self.mask_logits, (1, 1, 4, 4), "mask_logits")
        check_tensor(self.conditioned, (1, 4, 2, 2), "conditioned")


@dataclass
class FrameState:
    """Image features are unconditioned; propagated outputs are derived state."""

    cached_features: dict = field(default_factory=dict)
    conditioning: dict = field(default_factory=dict)
    nonconditioning: dict = field(default_factory=dict)

    def cache(self, index, features):
        """Store a synthetic unconditioned image embedding, independently of masks."""
        check_index(index)
        check_tensor(features, (1, 4, 2, 2), "features")
        self.cached_features[index] = features

    def store(self, index, record, prompted=False):
        """A conditioning output supersedes a derived output at the same frame."""
        check_index(index)
        if not isinstance(record, FrameRecord):
            raise ValueError("record must be a FrameRecord")
        if prompted:
            self.conditioning[index] = record
            self.nonconditioning.pop(index, None)
        elif index in self.conditioning:
            raise ValueError(
                "cannot overwrite a conditioning frame by propagation"
            )
        else:
            self.nonconditioning[index] = record

2. Select prompted memories and recent traversal slots

Original paper

Paper excerpt · PDF p. 5Official _prepare_memory_conditioned_features

The release selects temporally closest conditioning frames, then searches recent nonconditioning slots. A conditioning frame excluded from the closest set can fill a recent slot instead.

Math and tensor shapes

N=4,\quad j\in\{1,2,3\}
t_j=t+\delta(N-j)
\delta=-1\;\text{forward},\quad +1\;\text{reverse}

N is num_maskmem; j is the original temporal slot index. The closest-conditioning policy reserves a before and an after frame when both exist, then fills remaining capacity by distance. Forward recent slots run oldest to newest; reverse traversal uses larger chronological indices that were processed earlier in that pass. This fixture fixes evaluation stride to one. Online mode rejects future/current unprompted records and excludes future prompts; offline interactive mode can use a future prompted spatial memory. Attention selection bounds do not evict the stored output dictionaries.

Up to 2 selected prompted records + 3 recent spatial slots

Python implementation

2. Select prompted memories and recent traversal slots
python
def select_memory(
    state,
    current,
    num_maskmem=4,
    max_conditioning=2,
    reverse=False,
    online=False,
):
    """Use the pinned closest-conditioning policy and stride-one recent slots.

    Online rejects future unprompted evidence. Reverse traversal is explicitly
    offline: its already processed memories have larger chronological indices.
    Storage is not evicted; these bounds apply to spatial attention inputs.
    """
    check_index(current)
    if (
        isinstance(num_maskmem, bool)
        or not isinstance(num_maskmem, int)
        or num_maskmem < 1
        or max_conditioning not in (-1,)
        and (not isinstance(max_conditioning, int) or max_conditioning < 2)
    ):
        raise ValueError(
            "require num_maskmem >= 1 and max_conditioning >= 2 or -1"
        )
    if online and reverse:
        raise ValueError("reverse traversal is offline")
    for index in (*state.conditioning, *state.nonconditioning):
        check_index(index)
    if online and any(index >= current for index in state.nonconditioning):
        raise ValueError(
            "online mode rejects future/current unprompted evidence"
        )
    eligible = {
        i: r
        for i, r in state.conditioning.items()
        if i != current and (not online or i < current)
    }
    if max_conditioning == -1 or len(eligible) <= max_conditioning:
        selected = dict(eligible)
    else:
        selected = {}
        before = max((i for i in eligible if i < current), default=None)
        after = min((i for i in eligible if i >= current), default=None)
        for index in (before, after):
            if index is not None:
                selected[index] = eligible[index]
        remaining = sorted(
            (i for i in eligible if i not in selected),
            key=lambda i: abs(i - current),
        )
        for index in remaining[: max_conditioning - len(selected)]:
            selected[index] = eligible[index]
    unselected = {i: r for i, r in eligible.items() if i not in selected}
    spatial = [(i, 0, record) for i, record in selected.items()]
    direction = 1 if reverse else -1
    for slot in range(1, num_maskmem):
        index = current + direction * (num_maskmem - slot)
        record = state.nonconditioning.get(index, unselected.get(index))
        if record is not None:
            spatial.append((index, slot, record))
    return spatial, selected, unselected

3. Add the original learned temporal slots

Original paper

Paper excerpt · PDF p. 6Official spatial-memory temporal embedding indexing

All selected conditioning memories use t_pos=0; recent memories use the traversal slots above. The original code adds a learned temporal table entry to the memory positional encoding.

Math and tensor shapes

j=0\;\text{for selected prompts}
K_j=M_j+E_{N-j-1}
V_j=M_j

M_j is the flattened spatial memory and E is a seeded stand-in for the learned temporal table. Prompted frames share E_3 rather than receiving a distance-specific temporal encoding. The newest recent slot j=3 uses E_0. Reversing traversal changes which chronological frame fills a slot, not this table indexing. The fixture omits spatial positional encodings and RoPE. Pointer temporal encoding is disabled to follow the original-release convention, rather than importing SAM 2.1 signed-distance options.

Each [1,4,2,2] map → 4 tokens of width 4; E [4,4]

Python implementation

3. Add the original learned temporal slots
python
def time_positions(spatial, temporal_table, num_maskmem):
    """Apply original learned slot indexing; prompted frames share one slot.

    No signed-distance pointer positions are used: pointer temporal encoding is
    disabled in this original-release convention, unlike later SAM 2.1 options.
    Spatial sinusoidal/RoPE positions are omitted in this reduced fixture.
    """
    check_tensor(temporal_table, (num_maskmem, 4), "temporal_table")
    values, keys = [], []
    for _, slot, record in spatial:
        if not 0 <= slot < num_maskmem:
            raise ValueError("temporal slot outside bank")
        tokens = record.spatial.flatten(2).transpose(1, 2).squeeze(0)
        values.append(tokens)
        keys.append(tokens + temporal_table[num_maskmem - slot - 1])
    empty = temporal_table.new_empty(0, 4)
    return (
        torch.cat(keys) if keys else empty,
        torch.cat(values) if values else empty,
    )

4. Query spatial tokens and distinct pointer tokens

Original paper

Paper excerpt · PDF p. 17Official cross-attention operation · Appendix D.1

Memory attention uses current image tokens as queries and retained memory as keys/values. The actual stack also has self-attention and an MLP; pointer tokens have special handling in RoPE attention.

Math and tensor shapes

Q\in\mathbb R^{4\times4}
A=\operatorname{softmax}(QK^T/\sqrt4)
Y=Q+AV
\mathcal M=\varnothing\Rightarrow Y=Q+e_{\rm no\,mem}

K concatenates temporally positioned spatial tokens with split pointer tokens; V concatenates unpositioned spatial tokens and pointers. This one-head identity-projection residual demonstrates cross-attention algebra only. It is not the trained self/cross-attention/MLP stack, and has no mask decoder. For an empty initial bank it directly adds a seeded no-memory embedding, matching one original initial-conditioning branch. The generalized empty-bank fallback is a fixture convenience; later real propagation expects a conditioning frame.

4 current tokens attend to 4 per spatial memory + 2 per pointer

Python implementation

4. Query spatial tokens and distinct pointer tokens
python
def memory_attention(features, keys, values, pointer_tokens, no_memory):
    """One reduced cross-attention residual; no decoder and no tracking output."""
    check_tensor(features, (1, 4, 2, 2), "features")
    check_tensor(no_memory, (4,), "no_memory")
    for name, tokens in (
        ("keys", keys),
        ("values", values),
        ("pointer_tokens", pointer_tokens),
    ):
        if not isinstance(tokens, torch.Tensor) or tokens.ndim != 2:
            raise ValueError(f"{name} must be a [tokens,4] tensor")
        check_tensor(tokens, (tokens.shape[0], 4), name)
    if keys.shape != values.shape:
        raise ValueError("keys and values must have matching shape")
    query = features.flatten(2).transpose(1, 2).squeeze(0)
    if keys.shape[0] + pointer_tokens.shape[0] == 0:
        # Matches the directly_add_no_mem_embed initial-frame branch.
        output = query + no_memory
    else:
        key = torch.cat((keys, pointer_tokens))
        value = torch.cat((values, pointer_tokens))
        weights = torch.softmax(query @ key.T / math.sqrt(4), dim=-1)
        output = query + weights @ value
    return output.T.reshape(1, 4, 2, 2)

5. Fuse supplied mask logits with unconditioned features

Original paper

Paper excerpt · PDF p. 5Official _encode_new_memory · memory_encoder.py

The original encoder takes unconditioned image features and mask information. The release applies a sigmoid or an evaluation-time hard point-mask choice, then mask scale/bias before downsampling and fusion.

Math and tensor shapes

p=\sigma(L)\;\text{or}\;\mathbf1[L>0]
\widetilde p=20p-10
U=P(I)+D(\widetilde p)
M=\operatorname{Fuse}(U)

L is a supplied mask-logit fixture; P is a 1×1 feature projection, D a 2×2 stride-two mask convolution, and Fuse a 3×3 convolution. The original modules have deeper downsampling and convolutional fusion plus positional encoding; these trainable reduced modules expose the same data dependency. The (20,−10) convention is from the original release configuration, not a paper equation. from_points explicitly models the hard evaluation branch. Synthetic gradients verify fusion algebra, not learned segmentation.

Pinned scale/bias and pointer configuration

Mask [1,1,4,4] → downsample [1,4,2,2] + projected features

Python implementation

5. Fuse supplied mask logits with unconditioned features
python
class ReducedMemoryEncoder(nn.Module):
    """Reduced trainable mask downsampling, feature sum and convolutional fusion."""

    def __init__(self):
        super().__init__()
        self.mask_downsample = nn.Conv2d(1, 4, 2, stride=2)
        self.feature_projection = nn.Conv2d(4, 4, 1)
        self.fuser = nn.Conv2d(4, 4, 3, padding=1)

    def forward(self, features, mask_logits, from_points=False):
        """Follow sigmoid scale/bias (20,-10) or hard point-mask convention."""
        check_tensor(features, (1, 4, 2, 2), "features")
        check_tensor(mask_logits, (1, 1, 4, 4), "mask_logits")
        probabilities = (
            (mask_logits > 0).float() if from_points else mask_logits.sigmoid()
        )
        scaled_mask = probabilities * 20.0 - 10.0
        fused = self.feature_projection(features) + self.mask_downsample(
            scaled_mask
        )
        return self.fuser(fused)

6. Keep decoder pointers and absence separate from spatial memory

Original paper

Paper excerpt · PDF p. 17Official pointer selection and splitting

Pointers come from mask-decoder output tokens and are selected independently of spatial memory slots. With object-score prediction enabled, nonpositive scores suppress masks and can substitute a fixed no-object pointer.

Math and tensor shapes

z>0\Rightarrow L=L_{\rm fixture}
z>0\Rightarrow p=p_{\rm fixture}
z\leq0\Rightarrow L=-1024
z\leq0\Rightarrow p=p_{\rm no\,obj}
p\in\mathbb R^8\longrightarrow2\times\mathbb R^4

z is a supplied presence logit; p is a synthetic decoder-token fixture, never a mean of spatial features. The absence case cannot copy a previous visible mask. The reduced pointer split preserves the original 256→4×64 idea at smaller widths. Selected conditioning pointers plus up to max_pointers−1 recent offsets are searched separately, so max_pointers is not a universal total-pointer FIFO cap. Original evaluation can restrict pointers to the traversal past even when future prompted spatial memory is allowed. Presence scoring supports absence but does not prove occlusion recovery.

Pinned mask and no-object pointer gate

Supplied decoder token [1,8] → two pointer tokens [2,4]

Python implementation

6. Keep decoder pointers and absence separate from spatial memory
python
def object_pointer(decoder_token, mask_logits, presence_logit, no_object):
    """Use supplied scores/tokens, hard absence and fixed no-object pointer.

    The token is a synthetic decoder-output fixture, never inferred from a
    spatial-memory average. Original pred_obj_scores/fixed_no_obj_ptr behavior
    gates mask logits at score > 0 and replaces an absent pointer.
    """
    check_tensor(decoder_token, (1, 8), "decoder_token")
    check_tensor(mask_logits, (1, 1, 4, 4), "mask_logits")
    check_tensor(no_object, (1, 8), "no_object")
    if not isinstance(presence_logit, (int, float)) or not math.isfinite(
        presence_logit
    ):
        raise ValueError("presence_logit must be finite")
    present = presence_logit > 0
    mask = (
        mask_logits
        if present
        else torch.full_like(mask_logits, NO_OBJECT_SCORE)
    )
    pointer = decoder_token if present else no_object
    return mask, pointer


def pointer_memory(
    state,
    current,
    selected,
    unselected,
    max_pointers=6,
    reverse=False,
    only_past=True,
):
    """Select pointers independently of spatial capacity, then split 8 into 2×4.

    Like the pinned code, the recent loop has max_pointers-1 offsets; selected
    conditioning pointers are added separately, so this is not a universal
    total-pointer FIFO cap. Caller supplies a video-length-clamped limit.
    """
    check_index(current)
    if (
        isinstance(max_pointers, bool)
        or not isinstance(max_pointers, int)
        or max_pointers < 1
    ):
        raise ValueError("max_pointers must be a positive integer")
    direction = 1 if reverse else -1
    pointers = [
        r.pointer
        for i, r in selected.items()
        if not only_past or (i >= current if reverse else i <= current)
    ]
    for distance in range(1, max_pointers):
        index = current + direction * distance
        if index < 0:
            break
        record = state.nonconditioning.get(index, unselected.get(index))
        if record is not None:
            pointers.append(record.pointer)
    return (
        torch.cat(pointers).reshape(-1, 4)
        if pointers
        else torch.empty(0, 4, dtype=torch.float32)
    )

7. Replace a prompt and invalidate derived propagation

Original paper

Paper excerpt · PDF p. 5Official optional surrounding-memory clearing · inference lifecycle

Corrections change conditioning outputs, so derived propagation can become stale. The original predictor offers optional local nonconditioning-memory clearing around the new input, while image embeddings remain unconditioned.

Math and tensor shapes

\mathcal C[t]\leftarrow R_t^{\rm corrected}
\mathcal U\leftarrow\varnothing
I_s^{\rm cached}\;\text{remains reusable}

This fixture conservatively clears all derived nonconditioning records, including reverse-pass records, instead of reproducing the predictor’s configurable local window. Other prompted records stay intact. Re-running attention changes the derived context; new masks must still be supplied explicitly because the reference has no segmentation decoder. The smoke compares before/after context and feature object identities. Cache reuse is not permission to reuse masks predicted under old conditioning.

Correct frame 0 → invalidate derived records 1,2,3; retain image cache

Python implementation

7. Replace a prompt and invalidate derived propagation
python
def correct(state, index, replacement):
    """Replace a prompted output and conservatively invalidate derived records.

    This fixture clears ALL nonconditioning outputs, including reverse passes:
    a stricter lifecycle than the pinned predictor's optional local clearing.
    Unconditioned image features and other prompted outputs remain reusable.
    No old masks are silently accepted as recomputed predictions.
    """
    state.store(index, replacement, prompted=True)
    invalidated = sorted(state.nonconditioning)
    state.nonconditioning.clear()
    return invalidated

Smoke JSON records source SHA-256, upstream revision, seed, reduced shapes, selected frames, pointer count, absent-mask suppression, invalidated records, cache reuse, correction context change and finite fusion gradients. Tests compare attention values and derivatives with an independent scalar oracle. These are mechanics observations from fixtures, not pretrained-model outputs.

sam2.pyrequirements-sam2.txtOriginal paper §4Original Appendix D.1Pinned closest-frame helperPaper v2 · SAM 2.1 results

python -m pip install -r requirements-sam2.txt
python sam2.py --smoke --seed 7

Surviving occlusion

Video object segmentation has a hard failure mode: occlusion. When another object passes in front of the target — or the target moves behind a wall, dips below a waterline, or exits the frame entirely — there is no mask to produce. A naive model might hallucinate a mask on whatever is nearby, or collapse to an empty prediction that never recovers.

SAM 2 adds an object-presence head to the mask decoder. In the original release, a nonpositive object-score logit suppresses mask logits with a no-object sentinel, and the configured pointer path can substitute a learned no-object pointer. Memory encoding still uses the current frame’s unconditioned image features and the gated mask. This supports absence without copying an old visible mask.

The following reappearance sequence is a hand-authored schematic with supplied presence scores, not pretrained SAM 2 output or a calibrated probability experiment.

When the trained decoder finds the object again, retained prompted and recent context can help it predict a new mask. This is a learned segmentation outcome, not something guaranteed by a score threshold or proven by the schematic. The paper’s qualitative examples and limitations discuss behavior on occlusion and difficult scenes. The reference above stops at attention-conditioned features and memory encoding.

Why it mattered

SAM 2 was the first model to offer promptable segmentation for both images and video in a single architecture, running at interactive speeds (around 44 frames per second on an A100). Prior video object segmentation models required per-object fine-tuning or were limited to specific camera settings; SAM 2 works zero-shot across surveillance, sports, medical, and natural-video footage.

The streaming memory design is practical for deployment: the spatial attention window and pointer history are bounded. Attention cost therefore need not grow quadratically with clip length. This does not imply constant total application memory: storing a clip and its propagated outputs can still grow with the video length.

The SA-V dataset released alongside the model has become a standard benchmark for video segmentation research, and the open-sourced model weights have been integrated into annotation tools, robotics pipelines, and video editing workflows in much the same way that original SAM reshaped image annotation.

  • Segment Anything (SAM) — the image-only predecessor whose promptable segmentation task and mask decoder architecture SAM 2 directly extends
  • DETR — transformer-based detection that introduced learned object queries, an idea related to SAM 2’s object-pointer embeddings
  • CLIP — vision-language alignment that established the foundation-model paradigm for visual understanding that both SAM and SAM 2 follow

If you found this paper review helpful, consider sharing it with others.

Mastodon