SAM Video NodeSet

The SAM Video NodeSet (SamVideoNodeSet, workspace/nodesets/model/model_sam_video.py) wraps SAM 2 video object tracking via transformers Sam2VideoModel: prompt an object once on the first frame (a click or a box) and SAM 2 propagates a mask for that object across every subsequent frame. It is the temporal counterpart to SAM's per-frame image segmentation — "keep this object segmented as I walk" rather than "segment this frame". Following an instructed landmark through an egocentric rollout, persisting an object identity across steps, and mask-based visual servoing all want this.

env: ac-fm (shared FM env) · backend: transformers Sam2VideoModel + Sam2VideoProcessor · default: facebook/sam2.1-hiera-tiny (ungated)

1. Why a separate nodeset (not a model_sam variant)

model_sam is a fully stateless image server, with an explicit ruling that video/session state does not belong on it. This nodeset honours that. The SAM 2 inference session — the object memory that makes tracking work — is created, prompted, propagated, and torn down entirely inside one tool call, over a whole clip. Nothing survives across calls; the session is a call-local temporary, exactly like any other forward's scratch state. The server stays stateless, and the "segment a frame" and "track across a clip" capabilities live in their own files.

2. Frames in, prompt on frame 0

The frames port accepts either a JSON list of base64 PNG/JPEG frames, or a base64-npy {"shape":[T,H,W,3],"dtype":"uint8","b64":…} buffer. Prompt the object on frame 0 with either:

• points — a JSON list of [x, y] pixel coordinates (foreground unless labels marks some background: 1=fg, 0=bg), or
• box — a JSON [x1, y1, x2, y2] box.

If both are given, the box wins. With no prompt the call degrades (there is nothing to track). Tracking is single-object (object id 1); multi-object tracking is left for a future contract.

3. Model & propagation

The engine opens a video session on the clip, registers the frame-0 prompt for object 1, then iterates SAM 2's propagator, which yields a mask for the tracked object on each frame; every frame's mask is upscaled back to the original resolution and written into the output stack:

session = processor.init_video_session(video=frames, inference_device=device)
# prompt object 1 once, on frame 0 (a click here; a box is symmetric)
processor.process_new_points_or_boxes_for_video_frame(
    session, frame_idx=0, obj_ids=[1], input_points=[[[[x, y]]]], input_labels=[[[1]]])
for seg in model.propagate_in_video_iterator(session, start_frame_idx=0):
    mask = processor.post_process_masks([seg.pred_masks], original_sizes=[[H, W]])[0]
    out[seg.frame_idx] = mask > 0                # (T, H, W) uint8, filled frame by frame

The checkpoint is a lazy singleton in a registry, frozen and moved to the device once; the whole track is single-flight per engine (one clip in flight bounds peak VRAM under concurrent workers).

4. Masks envelope

The output is a TEXT JSON envelope — the per-frame masks as a single raw C-contiguous uint8 buffer, base64-encoded:

{"shape": [32, 480, 640], "dtype": "uint8", "b64": "<buffer>", "variant": "facebook/sam2.1-hiera-tiny", "object_id": 1}

masks[t] is the tracked object's binary mask ({0, 1}) on frame t, at the original resolution. Frames where the object is absent or fully occluded come back all-zero.

5. Canvas Node

model_sam_video__track

6. Server Mode

parallelism = "shared" (the inference session is call-local, so the server holds no cross-call state — one server across eval workers); server_python = conda_env_python("ac-fm", "SAM_VIDEO_PYTHON"). SAM 2 video runs in the shared ac-fm env — Sam2VideoModel is native to the transformers stack there. SAM 3 video (text-concept tracking, gated weights) is a future variant, deferred until a consumer needs it. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_sam_video/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, malformed/mixed-size frames, or a missing prompt yields "" on the node's output. An empty envelope is a clean "no track this clip" signal; consumers keep their own fallback and never receive a fabricated mask stack.