SAM NodeSet

The SAM NodeSet (SamNodeSet, workspace/nodesets/model/model_sam.py) wraps the Segment Anything family — SAM 1, SAM 2.1, and SAM 3 as real backends — into pure single-step segmentation primitives: point/box/auto/text segmentation plus an image-embedding node. Rewritten 2026-07-05 to the FM-nodeset template: variant and checkpoint are node config (swapping them never changes the graph — every variant returns the same masks envelope), and the server is fully stateless: no embedding cache, no sessions. Everything procedural — iterative refinement, embedding reuse, cross-model composition — lives in the graph, not in the nodeset. SAM 2.1/3 are served through the resident transformers (≥ 5.13 ships Sam2Model/Sam3Model), so the whole family costs zero extra dependencies.

env: ac-fm

Embedding in/out (the envelope ports) is sam1-only for now: SAM 2/3 image features are multi-level, which the single-tensor envelope cannot carry (an envelope v2 is a separate decision). Prompted nodes configured to sam2 emit an empty image_embedding and raise on envelope injection.

1. Canvas Nodes

model_sam__segment_points

Iterative refinement belongs to the outer loop — the node is a pure single-step forward. Each candidate carries low_res_logits_b64 (256×256 float32 ≈ 256 KB); wiring one back into mask_input closes the refinement cycle at graph level, keeping pure dataflow and episode replayability.

model_sam__segment_box

model_sam__segment_auto

Deviations on the sam2 path: the HF pipeline does not expose per-candidate stability_score (the key is simply absent from sam2 auto candidates), min_mask_area is applied as a local post-filter, and crop_n_layers > 0 raises — the transformers 5.13 pipeline crashes stacking unequal-sized crops (upstream bug), and a clear node error beats silently ignoring the knob. Use sam1 for cropped auto-segmentation.

model_sam__segment_text

This is SAM 3's native concept prompting — no detector in front. The weights are HF-gated (facebook/sam3, manual approval on the model page); until access is granted the engine latches degraded (§5). The graph-level GDINO → __segment_box composition remains the text route for the sam1/sam2 variants.

model_sam__embed_image

Reuse is a graph-level decision. SAM's structure is a heavy image encoder + light prompt decoder; the wrapper exposes that split as ports instead of a server-side cache. A graph that prompts the same frame repeatedly runs __embed_image once (or catches any prompted node's image_embedding output), stashes the envelope in a state container, and feeds it back into the image port — the encoder is skipped, and the server stays a pure function. SAM 1's envelope is ≈ 5.6 MB of TEXT (1×256×64×64 float32, base64), which the msgpack transport handles comfortably.

2. Envelopes

All segmentation nodes return the same masks envelope regardless of variant — swapping checkpoints never changes the graph:

{"masks": [{"mask_index": 0,
            "mask_b64": "<base64 PNG, 0/255 grayscale>",
            "bbox_xyxy": [201, 121, 439, 359],
            "iou_score": 0.9772,
            "area": 44945,
            "low_res_logits_b64": "<base64 of 256x256 float32 buffer>"}],
 "count": 3, "image_w": 640, "image_h": 480}

low_res_logits_b64 appears on prompted candidates only (__segment_auto discards logits); sam1 auto candidates carry stability_score instead (the sam2 pipeline does not expose it); __segment_text candidates reuse iou_score for the SAM 3 instance score. The mask_b64 key name is load-bearing: aoplanner's _union_sam_masks parses it.

The embedding envelope is self-describing — a byte-exact float32 buffer plus the predictor sizes and the provenance needed to reject a mismatched injection:

{"b64": "<base64 of the C-contiguous float32 buffer>",
 "shape": [1, 256, 64, 64], "dtype": "float32",
 "original_hw": [480, 640], "input_hw": [768, 1024],
 "variant": "sam1", "ckpt_id": "sam_vit_b.pth"}

Injecting an envelope whose variant/ckpt_id mismatches the target node's config raises — a node error beats a silently wrong mask. Injection round-trips byte-identically: masks produced from an injected envelope equal the direct-image path exactly (verified through the server transport as well).

3. Server Mode

parallelism = "shared"; server_python = conda_env_python("ac-fm", "SAM_PYTHON") since 2026-07-05 (segment-anything is lower-bound-only → shared-env admission; installed with opencv-python-headless by scripts/install/install_ac_fm.sh). One engine class per variant (_Sam1Engine/_Sam2Engine/_Sam3Engine) behind a lazy registry keyed (variant, ckpt) — several checkpoints coexist in one server — and engines hold nothing but loaded weights: no cache, no sessions, every call builds a fresh predictor. The GPU section is single-flight per engine to bound VRAM under concurrent eval workers. SAM 2.1/3 use the transformers implementations, not the facebookresearch packages (nothing new to install; the PyPI sam2/sam3 names have uncertain provenance). Loading the sam2.1 checkpoint logs a benign sam2_video-vs-sam2 config-type warning — the image path shares a weight subset, expected per HF.

Verification: the sam1 rewrite is byte-faithful — on the old hosting env, point/box/auto outputs are byte-identical to the pre-rewrite code (auto compared at its effective old grid, see the no-op knob note in §1). Serving from ac-fm shifts torch 2.5.1 → 2.8.0, whose conv kernels drift mask boundaries by ±1 px and IoU scores at the third decimal — accepted, since no graph or test consumed the old outputs. The sam2/sam3 variants have no byte baseline (different implementation lineage), so they gate on synthetic-image analytic-GT IoU ≥ 0.9 (point/box/auto/text all ≥ 0.97 at verification, matching a 0.99 sam1 calibration), determinism, and a logits→mask_input round-trip. Load: POST /api/components/nodesets/model_sam/load?mode=server.

4. Environment Variables

The former SAM_VERSION / SAM_MODEL_CFG env vars are gone — variant and checkpoint are node config (env panels carry env-side runtime knobs only; model identity belongs on the node). Per-variant checkpoint defaults when ckpt is blank:

5. Degraded Mode

An engine load failure latches (no retry storm) and every call on that engine returns empty outputs (masks = "", image_embedding = "") with a degraded self-log entry. Input-contract violations — malformed JSON, an embedding envelope for the wrong variant/ckpt — raise instead: the executor records a node error and the eval layer convicts the episode.

6. Composition & Consumers

• Text-prompted segmentation has two routes: the native __segment_text (SAM 3 concept prompting) and the graph composition wiring model_grounding_dino__detect boxes into __segment_box — the latter is the path when sam3's gated weights are unavailable or a sam1/sam2 mask is wanted.
• Video tracking stays deferred even with SAM 2 landed — session-style capabilities need explicit handles, which no current graph justifies.
• First graph consumer (2026-07-06): aoplanner_ce wires __segment_box (sam1, ViT-H ckpt) per panorama lane — model_grounding_dino__detect boxes → aoplanner__ground_boxes → __segment_box → sample_waypoints.sam_result (union via _union_sam_masks, which reads masks[].mask_b64 — the key name is load-bearing). This replaced the Grounded-SAM that was embedded in the deleted model_grounding_dino__ground_mask.