TL;DR
- SAM 2 generalizes SAM’s promptable segmentation from single images to video: a single prompt on one frame propagates the mask across all subsequent frames without additional user input.
- A streaming memory bank stores spatial memories and separate object pointers. Memory attention conditions the current frame on selected prompted frames and recent predictions; interactive propagation can also use a future prompted frame.
- An object-presence head supports absent-object outputs. Stored context can help a trained model recover after occlusion, but a presence score does not guarantee recovery.
- SAM 2 unifies image and video segmentation in a single real-time model, trained on the SA-V dataset (50.9k videos, 35.5M masks), and achieves strong zero-shot video object segmentation across multiple benchmarks.
From images to video
SAM defined promptable image segmentation: given a point, box, or mask prompt, produce a valid segmentation mask for any object. The model is remarkably general, but it has no notion of time — each image is processed in isolation.
SAM 2 extends this to video. Online processing consumes available frames without future evidence. Offline interactive propagation lets the user prompt or correct any frame in a recorded clip and run forward or backward; a prompted memory can then lie in the chronological future of the current frame. For image segmentation, SAM 2 simply runs the same pipeline on a single-frame clip, so images and videos share a single unified model.
The architectural change that makes this possible is a streaming memory bank paired with a memory attention module. Instead of discarding per-frame computation after producing a mask, SAM 2 stores a compact representation of each processed frame. When processing the next frame, it retrieves these stored memories and uses cross-attention to condition the current frame’s features on the object’s history. This gives the model a persistent, temporally coherent view of the object across the clip.
The model was trained on SA-V, a new video dataset built with a similar model-in-the-loop annotation engine to SA-1B, containing 50.9k videos and 35.5 million masks — by far the largest annotated video segmentation dataset. The backbone is Hiera, a hierarchical vision transformer that is more efficient than the ViT-H used in original SAM.
Streaming memory bank
Spatial memory comes from prompted conditioning frames and a bounded recent set of nonconditioning predictions. Both store mask-conditioned spatial feature maps. A separate list of object pointers, extracted from mask-decoder output tokens, supplies compact object summaries with its own selection horizon and token splitting.
The bank is not a universal FIFO of identical entries. The original code selects temporally closest conditioning frames, keeping before/after context when available, and then chooses recent traversal slots. Prompted memories are not simply discarded when a recent unprompted window fills. The illustration below fixes one prompt and forward traversal; its circles and masks are schematics, not model predictions.
For each frame, memory attention first conditions the current image features on selected stored memories, and the mask decoder predicts a mask from those conditioned features and any prompt. The memory encoder then fuses the unconditioned image features with the predicted mask to create a spatial memory for later frames. In online mode the system can run without future frames. Offline interactive mode may attend to future prompted spatial memories, while evaluation can still restrict object pointers to the traversal past.
Memory attention
Memory attention is the mechanism that connects the current frame to its stored history. It is a form of cross-attention: the current frame’s image features act as the queries, and the selected stored memory entries (prompted/recent spatial feature maps plus object-pointer embeddings) supply the keys and values.
A KV cache is a useful comparison for reusing context, but SAM 2 does not store a language decoder’s layer-by-layer key/value cache. It encodes masks with image features, selects spatial memories and decoder-derived pointers, and constructs the attention inputs with positional information. These representations have different limits and update rules; offline corrections can change the context and invalidate derived outputs.
Conditioning on more memory entries gives the model a richer view of the object’s appearance over time, which helps when the object changes scale, pose, or illumination across frames. In practice spatial memories and pointers follow distinct selection policies. Bounded attention inputs balance temporal context against compute; stored video frames, cached image features and output dictionaries have their own memory costs.
Implementation
The memory-mechanics reference below makes selection, absence and correction state explicit before returning to the trained model’s occlusion behavior. It uses supplied mask and pointer fixtures, so it cannot demonstrate segmentation or reappearance recovery.
Seeded offline CPU memory mechanics with synthetic image embeddings, supplied mask logits and supplied decoder tokens. Widths 4/8 and grids 2×2/4×4 are reduced; one cross-attention residual and three convolutions replace the trained modules. There is no Hiera, mask decoder, checkpoint, pretrained tracking, video segmentation, occlusion-recovery proof or benchmark reproduction. All equations are authored notation for the cited operations. The code follows the pinned August 2024 original SAM 2 release; paper v2 reports SAM 2.1 results and is not its code specification.
1. Separate image features from derived outputs
Original paper
The predictor caches unconditioned image features separately from conditioning and nonconditioning frame outputs. The latter include encoded memory and decoder pointers, not just image features.
Math and tensor shapes
I_t is a cached synthetic image embedding. C and U are separate dictionaries of FrameRecord values R_t. A prompted output supersedes the derived output at the same index. The fixture retains every supplied image embedding for correction tests; the pinned predictor uses a small cache and can recompute evicted embeddings. These inputs are fixtures, not Hiera outputs from a real video.
Record: spatial [1,4,2,2], pointer [1,8], mask logits [1,1,4,4]
Python implementation
@dataclass
class FrameRecord:
"""Keep supplied masks, encoded spatial memory and decoder pointers apart."""
spatial: torch.Tensor # [1,4,2,2]
pointer: torch.Tensor # [1,8], split into two width-4 attention tokens
mask_logits: torch.Tensor # [1,1,4,4], absence is -1024
conditioned: torch.Tensor # [1,4,2,2], derived from current bank
def __post_init__(self):
check_tensor(self.spatial, (1, 4, 2, 2), "spatial")
check_tensor(self.pointer, (1, 8), "pointer")
check_tensor(self.mask_logits, (1, 1, 4, 4), "mask_logits")
check_tensor(self.conditioned, (1, 4, 2, 2), "conditioned")
@dataclass
class FrameState:
"""Image features are unconditioned; propagated outputs are derived state."""
cached_features: dict = field(default_factory=dict)
conditioning: dict = field(default_factory=dict)
nonconditioning: dict = field(default_factory=dict)
def cache(self, index, features):
"""Store a synthetic unconditioned image embedding, independently of masks."""
check_index(index)
check_tensor(features, (1, 4, 2, 2), "features")
self.cached_features[index] = features
def store(self, index, record, prompted=False):
"""A conditioning output supersedes a derived output at the same frame."""
check_index(index)
if not isinstance(record, FrameRecord):
raise ValueError("record must be a FrameRecord")
if prompted:
self.conditioning[index] = record
self.nonconditioning.pop(index, None)
elif index in self.conditioning:
raise ValueError(
"cannot overwrite a conditioning frame by propagation"
)
else:
self.nonconditioning[index] = record2. Select prompted memories and recent traversal slots
Original paper
The release selects temporally closest conditioning frames, then searches recent nonconditioning slots. A conditioning frame excluded from the closest set can fill a recent slot instead.
Math and tensor shapes
N is num_maskmem; j is the original temporal slot index. The closest-conditioning policy reserves a before and an after frame when both exist, then fills remaining capacity by distance. Forward recent slots run oldest to newest; reverse traversal uses larger chronological indices that were processed earlier in that pass. This fixture fixes evaluation stride to one. Online mode rejects future/current unprompted records and excludes future prompts; offline interactive mode can use a future prompted spatial memory. Attention selection bounds do not evict the stored output dictionaries.
Up to 2 selected prompted records + 3 recent spatial slots
Python implementation
def select_memory(
state,
current,
num_maskmem=4,
max_conditioning=2,
reverse=False,
online=False,
):
"""Use the pinned closest-conditioning policy and stride-one recent slots.
Online rejects future unprompted evidence. Reverse traversal is explicitly
offline: its already processed memories have larger chronological indices.
Storage is not evicted; these bounds apply to spatial attention inputs.
"""
check_index(current)
if (
isinstance(num_maskmem, bool)
or not isinstance(num_maskmem, int)
or num_maskmem < 1
or max_conditioning not in (-1,)
and (not isinstance(max_conditioning, int) or max_conditioning < 2)
):
raise ValueError(
"require num_maskmem >= 1 and max_conditioning >= 2 or -1"
)
if online and reverse:
raise ValueError("reverse traversal is offline")
for index in (*state.conditioning, *state.nonconditioning):
check_index(index)
if online and any(index >= current for index in state.nonconditioning):
raise ValueError(
"online mode rejects future/current unprompted evidence"
)
eligible = {
i: r
for i, r in state.conditioning.items()
if i != current and (not online or i < current)
}
if max_conditioning == -1 or len(eligible) <= max_conditioning:
selected = dict(eligible)
else:
selected = {}
before = max((i for i in eligible if i < current), default=None)
after = min((i for i in eligible if i >= current), default=None)
for index in (before, after):
if index is not None:
selected[index] = eligible[index]
remaining = sorted(
(i for i in eligible if i not in selected),
key=lambda i: abs(i - current),
)
for index in remaining[: max_conditioning - len(selected)]:
selected[index] = eligible[index]
unselected = {i: r for i, r in eligible.items() if i not in selected}
spatial = [(i, 0, record) for i, record in selected.items()]
direction = 1 if reverse else -1
for slot in range(1, num_maskmem):
index = current + direction * (num_maskmem - slot)
record = state.nonconditioning.get(index, unselected.get(index))
if record is not None:
spatial.append((index, slot, record))
return spatial, selected, unselected3. Add the original learned temporal slots
Original paper
All selected conditioning memories use t_pos=0; recent memories use the traversal slots above. The original code adds a learned temporal table entry to the memory positional encoding.
Math and tensor shapes
M_j is the flattened spatial memory and E is a seeded stand-in for the learned temporal table. Prompted frames share E_3 rather than receiving a distance-specific temporal encoding. The newest recent slot j=3 uses E_0. Reversing traversal changes which chronological frame fills a slot, not this table indexing. The fixture omits spatial positional encodings and RoPE. Pointer temporal encoding is disabled to follow the original-release convention, rather than importing SAM 2.1 signed-distance options.
Each [1,4,2,2] map → 4 tokens of width 4; E [4,4]
Python implementation
def time_positions(spatial, temporal_table, num_maskmem):
"""Apply original learned slot indexing; prompted frames share one slot.
No signed-distance pointer positions are used: pointer temporal encoding is
disabled in this original-release convention, unlike later SAM 2.1 options.
Spatial sinusoidal/RoPE positions are omitted in this reduced fixture.
"""
check_tensor(temporal_table, (num_maskmem, 4), "temporal_table")
values, keys = [], []
for _, slot, record in spatial:
if not 0 <= slot < num_maskmem:
raise ValueError("temporal slot outside bank")
tokens = record.spatial.flatten(2).transpose(1, 2).squeeze(0)
values.append(tokens)
keys.append(tokens + temporal_table[num_maskmem - slot - 1])
empty = temporal_table.new_empty(0, 4)
return (
torch.cat(keys) if keys else empty,
torch.cat(values) if values else empty,
)4. Query spatial tokens and distinct pointer tokens
Original paper
Memory attention uses current image tokens as queries and retained memory as keys/values. The actual stack also has self-attention and an MLP; pointer tokens have special handling in RoPE attention.
Math and tensor shapes
K concatenates temporally positioned spatial tokens with split pointer tokens; V concatenates unpositioned spatial tokens and pointers. This one-head identity-projection residual demonstrates cross-attention algebra only. It is not the trained self/cross-attention/MLP stack, and has no mask decoder. For an empty initial bank it directly adds a seeded no-memory embedding, matching one original initial-conditioning branch. The generalized empty-bank fallback is a fixture convenience; later real propagation expects a conditioning frame.
4 current tokens attend to 4 per spatial memory + 2 per pointer
Python implementation
def memory_attention(features, keys, values, pointer_tokens, no_memory):
"""One reduced cross-attention residual; no decoder and no tracking output."""
check_tensor(features, (1, 4, 2, 2), "features")
check_tensor(no_memory, (4,), "no_memory")
for name, tokens in (
("keys", keys),
("values", values),
("pointer_tokens", pointer_tokens),
):
if not isinstance(tokens, torch.Tensor) or tokens.ndim != 2:
raise ValueError(f"{name} must be a [tokens,4] tensor")
check_tensor(tokens, (tokens.shape[0], 4), name)
if keys.shape != values.shape:
raise ValueError("keys and values must have matching shape")
query = features.flatten(2).transpose(1, 2).squeeze(0)
if keys.shape[0] + pointer_tokens.shape[0] == 0:
# Matches the directly_add_no_mem_embed initial-frame branch.
output = query + no_memory
else:
key = torch.cat((keys, pointer_tokens))
value = torch.cat((values, pointer_tokens))
weights = torch.softmax(query @ key.T / math.sqrt(4), dim=-1)
output = query + weights @ value
return output.T.reshape(1, 4, 2, 2)5. Fuse supplied mask logits with unconditioned features
Original paper
The original encoder takes unconditioned image features and mask information. The release applies a sigmoid or an evaluation-time hard point-mask choice, then mask scale/bias before downsampling and fusion.
Math and tensor shapes
L is a supplied mask-logit fixture; P is a 1×1 feature projection, D a 2×2 stride-two mask convolution, and Fuse a 3×3 convolution. The original modules have deeper downsampling and convolutional fusion plus positional encoding; these trainable reduced modules expose the same data dependency. The (20,−10) convention is from the original release configuration, not a paper equation. from_points explicitly models the hard evaluation branch. Synthetic gradients verify fusion algebra, not learned segmentation.
Pinned scale/bias and pointer configuration
Mask [1,1,4,4] → downsample [1,4,2,2] + projected features
Python implementation
class ReducedMemoryEncoder(nn.Module):
"""Reduced trainable mask downsampling, feature sum and convolutional fusion."""
def __init__(self):
super().__init__()
self.mask_downsample = nn.Conv2d(1, 4, 2, stride=2)
self.feature_projection = nn.Conv2d(4, 4, 1)
self.fuser = nn.Conv2d(4, 4, 3, padding=1)
def forward(self, features, mask_logits, from_points=False):
"""Follow sigmoid scale/bias (20,-10) or hard point-mask convention."""
check_tensor(features, (1, 4, 2, 2), "features")
check_tensor(mask_logits, (1, 1, 4, 4), "mask_logits")
probabilities = (
(mask_logits > 0).float() if from_points else mask_logits.sigmoid()
)
scaled_mask = probabilities * 20.0 - 10.0
fused = self.feature_projection(features) + self.mask_downsample(
scaled_mask
)
return self.fuser(fused)6. Keep decoder pointers and absence separate from spatial memory
Original paper
Pointers come from mask-decoder output tokens and are selected independently of spatial memory slots. With object-score prediction enabled, nonpositive scores suppress masks and can substitute a fixed no-object pointer.
Math and tensor shapes
z is a supplied presence logit; p is a synthetic decoder-token fixture, never a mean of spatial features. The absence case cannot copy a previous visible mask. The reduced pointer split preserves the original 256→4×64 idea at smaller widths. Selected conditioning pointers plus up to max_pointers−1 recent offsets are searched separately, so max_pointers is not a universal total-pointer FIFO cap. Original evaluation can restrict pointers to the traversal past even when future prompted spatial memory is allowed. Presence scoring supports absence but does not prove occlusion recovery.
Pinned mask and no-object pointer gate
Supplied decoder token [1,8] → two pointer tokens [2,4]
Python implementation
def object_pointer(decoder_token, mask_logits, presence_logit, no_object):
"""Use supplied scores/tokens, hard absence and fixed no-object pointer.
The token is a synthetic decoder-output fixture, never inferred from a
spatial-memory average. Original pred_obj_scores/fixed_no_obj_ptr behavior
gates mask logits at score > 0 and replaces an absent pointer.
"""
check_tensor(decoder_token, (1, 8), "decoder_token")
check_tensor(mask_logits, (1, 1, 4, 4), "mask_logits")
check_tensor(no_object, (1, 8), "no_object")
if not isinstance(presence_logit, (int, float)) or not math.isfinite(
presence_logit
):
raise ValueError("presence_logit must be finite")
present = presence_logit > 0
mask = (
mask_logits
if present
else torch.full_like(mask_logits, NO_OBJECT_SCORE)
)
pointer = decoder_token if present else no_object
return mask, pointer
def pointer_memory(
state,
current,
selected,
unselected,
max_pointers=6,
reverse=False,
only_past=True,
):
"""Select pointers independently of spatial capacity, then split 8 into 2×4.
Like the pinned code, the recent loop has max_pointers-1 offsets; selected
conditioning pointers are added separately, so this is not a universal
total-pointer FIFO cap. Caller supplies a video-length-clamped limit.
"""
check_index(current)
if (
isinstance(max_pointers, bool)
or not isinstance(max_pointers, int)
or max_pointers < 1
):
raise ValueError("max_pointers must be a positive integer")
direction = 1 if reverse else -1
pointers = [
r.pointer
for i, r in selected.items()
if not only_past or (i >= current if reverse else i <= current)
]
for distance in range(1, max_pointers):
index = current + direction * distance
if index < 0:
break
record = state.nonconditioning.get(index, unselected.get(index))
if record is not None:
pointers.append(record.pointer)
return (
torch.cat(pointers).reshape(-1, 4)
if pointers
else torch.empty(0, 4, dtype=torch.float32)
)7. Replace a prompt and invalidate derived propagation
Original paper
Corrections change conditioning outputs, so derived propagation can become stale. The original predictor offers optional local nonconditioning-memory clearing around the new input, while image embeddings remain unconditioned.
Math and tensor shapes
This fixture conservatively clears all derived nonconditioning records, including reverse-pass records, instead of reproducing the predictor’s configurable local window. Other prompted records stay intact. Re-running attention changes the derived context; new masks must still be supplied explicitly because the reference has no segmentation decoder. The smoke compares before/after context and feature object identities. Cache reuse is not permission to reuse masks predicted under old conditioning.
Correct frame 0 → invalidate derived records 1,2,3; retain image cache
Python implementation
def correct(state, index, replacement):
"""Replace a prompted output and conservatively invalidate derived records.
This fixture clears ALL nonconditioning outputs, including reverse passes:
a stricter lifecycle than the pinned predictor's optional local clearing.
Unconditioned image features and other prompted outputs remain reusable.
No old masks are silently accepted as recomputed predictions.
"""
state.store(index, replacement, prompted=True)
invalidated = sorted(state.nonconditioning)
state.nonconditioning.clear()
return invalidatedSmoke JSON records source SHA-256, upstream revision, seed, reduced shapes, selected frames, pointer count, absent-mask suppression, invalidated records, cache reuse, correction context change and finite fusion gradients. Tests compare attention values and derivatives with an independent scalar oracle. These are mechanics observations from fixtures, not pretrained-model outputs.
sam2.pyrequirements-sam2.txtOriginal paper §4Original Appendix D.1Pinned closest-frame helperPaper v2 · SAM 2.1 results
python -m pip install -r requirements-sam2.txt python sam2.py --smoke --seed 7
Surviving occlusion
Video object segmentation has a hard failure mode: occlusion. When another object passes in front of the target — or the target moves behind a wall, dips below a waterline, or exits the frame entirely — there is no mask to produce. A naive model might hallucinate a mask on whatever is nearby, or collapse to an empty prediction that never recovers.
SAM 2 adds an object-presence head to the mask decoder. In the original release, a nonpositive object-score logit suppresses mask logits with a no-object sentinel, and the configured pointer path can substitute a learned no-object pointer. Memory encoding still uses the current frame’s unconditioned image features and the gated mask. This supports absence without copying an old visible mask.
The following reappearance sequence is a hand-authored schematic with supplied presence scores, not pretrained SAM 2 output or a calibrated probability experiment.
When the trained decoder finds the object again, retained prompted and recent context can help it predict a new mask. This is a learned segmentation outcome, not something guaranteed by a score threshold or proven by the schematic. The paper’s qualitative examples and limitations discuss behavior on occlusion and difficult scenes. The reference above stops at attention-conditioned features and memory encoding.
Why it mattered
SAM 2 was the first model to offer promptable segmentation for both images and video in a single architecture, running at interactive speeds (around 44 frames per second on an A100). Prior video object segmentation models required per-object fine-tuning or were limited to specific camera settings; SAM 2 works zero-shot across surveillance, sports, medical, and natural-video footage.
The streaming memory design is practical for deployment: the spatial attention window and pointer history are bounded. Attention cost therefore need not grow quadratically with clip length. This does not imply constant total application memory: storing a clip and its propagated outputs can still grow with the video length.
The SA-V dataset released alongside the model has become a standard benchmark for video segmentation research, and the open-sourced model weights have been integrated into annotation tools, robotics pipelines, and video editing workflows in much the same way that original SAM reshaped image annotation.
Related Reading
- Segment Anything (SAM) — the image-only predecessor whose promptable segmentation task and mask decoder architecture SAM 2 directly extends
- DETR — transformer-based detection that introduced learned object queries, an idea related to SAM 2’s object-pointer embeddings
- CLIP — vision-language alignment that established the foundation-model paradigm for visual understanding that both SAM and SAM 2 follow
