Segmentation NodeSet
The Segmentation NodeSet (SegmentationNodeSet, workspace/nodesets/model/model_segmentation.py) wraps Mask2Former universal segmentation via transformers. Where SAM answers "where are the objects" with class-agnostic masks, Mask2Former answers "what is each pixel": a per-pixel semantic label map, or a panoptic map that also separates instances. Semantic maps for VLN, walkable-vs-obstacle reasoning, and grounding a class name to image regions all build on this.
env: ac-fm (shared FM env) · backend: transformers Mask2FormerForUniversalSegmentation + AutoImageProcessor · defaults: …-swin-tiny-ade-semantic / …-swin-tiny-coco-panoptic (ungated)
| Primitive | Purpose |
|---|---|
model_segmentation__semantic |
Per-pixel semantic class-id map → segmentation (JSON envelope of (N, H, W) int32 + id2label). |
model_segmentation__panoptic |
Instance-aware segment-id map + per-segment class list → segmentation (JSON envelope of (N, H, W) int32 + segments). |
1. Two primitives: semantic vs panoptic
semantic labels every pixel with a class id (from a semantic checkpoint — ADE20K's 150 classes by default), naming each id via id2label. panoptic goes further: it separates instances, so each pixel carries a segment id, and a segments list gives each segment's class + score (from a panoptic checkpoint — COCO by default). A graph wires whichever fits — a semantic map for "which pixels are floor", panoptic for "how many distinct chairs". They are distinct weights, so each maps to its own engine instance.
2. Images in — uniform resolution
Both tools take the images port — a list of {rgb_base64} dicts (or raw base64 strings). All images in one call must share resolution (the envelope is a single (N, H, W) buffer); mixed sizes degrade to empty with a self-log. Maps are returned at the original input resolution — the processor's post_process_* resizes them back.
3. Envelopes
Each output is a TEXT JSON envelope — the label/segment map as a raw C-contiguous int32 buffer, base64-encoded (byte-exact across the server-mode HTTP boundary). Semantic carries id2label; panoptic carries a per-image segments list:
# semantic {"shape": [1, 480, 640], "dtype": "int32", "b64": "…", "id2label": {"0": "wall", …}} # panoptic {"shape": [1, 480, 640], "dtype": "int32", "b64": "…", "segments": [[{"id": 1, "label_id": 3, "label": "chair", "score": 0.98}, …]]}
In the semantic map each pixel value indexes id2label; in the panoptic map each pixel value is a segment id matched by the segments entries. In an agent loop N is typically 1 (per-frame parse).
4. Canvas Nodes
model_segmentation__semantic
| Field | Detail |
|---|---|
| Inputs | images (ANY — list of {rgb_base64} dicts or raw base64 strings, uniform size) |
| Outputs | segmentation (TEXT — (N,H,W) int32 + id2label; "" on degraded) |
| Config | model_id (text, default facebook/mask2former-swin-tiny-ade-semantic) |
| Backend call | processor → model(**pp) → post_process_semantic_segmentation |
model_segmentation__panoptic
| Field | Detail |
|---|---|
| Inputs | images (ANY — as above) |
| Outputs | segmentation (TEXT — (N,H,W) int32 + segments; "" on degraded) |
| Config | model_id (text, default facebook/mask2former-swin-tiny-coco-panoptic) |
| Backend call | processor → model(**pp) → post_process_panoptic_segmentation |
Engines are lazy singletons in a registry keyed by model_id (a semantic and a panoptic checkpoint are distinct weights → distinct engines); GPU inference is single-flight per engine.
5. Server Mode
parallelism = "shared" (a stateless segmenter — one server across eval workers); server_python = conda_env_python("ac-fm", "SEGMENTATION_PYTHON"). Segmentation runs in the shared ac-fm env — Mask2Former is native to the transformers stack there. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_segmentation/load?mode=server.
6. Environment Variables
| Variable | Default | Purpose |
|---|---|---|
SEGMENTATION_PYTHON | ac-fm env python | Interpreter for the server subprocess |
SEGMENTATION_DEVICE | auto (→ cuda when available) | Torch device for the loaded model |
7. Degraded Mode
A weight-load failure latches (_load_failed — no retry storm); that, a missing/malformed frame, or mixed input resolutions yields "" on the node's output. An empty envelope is a clean "no segmentation this step" signal; consumers keep their own fallback and never receive a fabricated label map.