Everyone's seen the SAM 2 demo clip by now: click one object in frame one, watch the mask follow it through the whole clip, occlusions and all. It's an impressive demo. What it doesn't show is what happens on frame 340 of a 400-frame clip when your subject walks behind a pole, comes out the other side wearing different lighting, and the memory bank has to decide whether that's still the same object.

I ran it against a set of clips from a warehouse camera project — people and pallets moving through a space with real occlusion, real lighting changes, real motion blur. Not adversarial, just ordinary footage. The propagation held up well for maybe 80% of tracks through the full clip length. The other 20% needed a manual re-click to recover after a hard occlusion, which is fine, that's what the interactive design is for, but it means "point-and-track" isn't quite "point once and forget it" for anything longer than a few seconds of genuinely messy video.

Where it's actually a big deal

The honest win isn't full autonomy, it's the cost of correction. With the old tracking-by-detection pipelines we used, recovering from a lost track meant re-running detection and re-associating, which is a few seconds of latency and sometimes a wrong association that silently corrupts the rest of the track. With SAM 2, recovering is a single click on the frame where it lost the object, and the memory bank picks the object back up from there. For an annotation pipeline — which is honestly the main thing we use it for now — that's the difference between a labeler spending two minutes per clip and twenty.

The memory attention mechanism is doing something closer to re-identification than tracking, and it shows: it's better at "is this the same object" than at "where exactly is the boundary of this object" during fast motion.

Boundary quality during fast motion is the other rough edge. Mask edges get soft and lag behind the actual object boundary for a few frames after a quick pan or a fast-moving subject. If your downstream task needs pixel-accurate boundaries on every frame — say, for a matting or rotoscoping use case — you'll still want a refinement pass. If you just need "which pixels are this object, roughly" for analytics or dataset generation, it's more than good enough.

Practical setup notes

  • Run it at the native frame rate you'll deploy at — we tested downsampling to save compute and lost track quality on fast subjects noticeably faster than the drop in frame rate would suggest.
  • The memory bank size is a real lever. The default is tuned for demo-length clips; for anything past a couple of minutes we had to bump it and eat the extra VRAM.
  • It's still worth pairing with a lightweight detector for automatic re-seeding after a lost track if you can't have a human in the loop — SAM 2 alone won't re-find an object it lost without a prompt.

Worth adopting if you're building an annotation tool or doing offline video analysis. Not yet what I'd reach for if you need a fully autonomous, no-human-in-the-loop tracker running live.