Pointmap NodeSet

The Pointmap NodeSet (PointmapNodeSet, workspace/nodesets/model/model_pointmap.py) wraps Sapiens single-image pointmap estimation via transformers Sapiens2ForPointmapEstimation: one RGB frame in, a dense per-pixel 3D (X, Y, Z) coordinate map out. It is the monocular counterpart to VGGT's multi-view pointmap — where VGGT fuses several views into one 3D field, this lifts a single image into 3D geometry directly, no intrinsics and no second view. Obstacle geometry, free-space extent, "how far is that wall", and back-projecting a 2D detection into a 3D point all want this.

env: ac-fm (shared FM env) · backend: transformers Sapiens2ForPointmapEstimation + AutoImageProcessor · default: facebook/sapiens2-pointmap-0.4b (ungated)

1. One primitive: RGB → 3D pointmap

The tool takes a list of same-resolution images and returns, per image, a per-pixel (X, Y, Z) coordinate in canonical camera space. It is the single-view sibling of Surface Normals and Depth Anything — same input contract, same envelope convention — so a monocular geometry trio (depth · normals · pointmap) composes as drop-in Sapiens/DINO nodes. In a rollout loop N is typically 1 (per-frame geometry).

2. Images in — uniform resolution

The images port accepts a list of {rgb_base64} dicts (or raw base64 strings). All images in one call must share resolution — the envelope is a single (N, H, W, 3) buffer, so mixed sizes degrade to empty with a self-log (split into uniform batches). A malformed frame degrades the whole call rather than guessing.

3. Model & post-processing

The engine loads Sapiens2ForPointmapEstimation + AutoImageProcessor for the configured model_id. A dedicated processor post-process crops the preprocessing padding and resizes each prediction back to its original (H, W), returning one {"pointmap": (3, H, W)} per image in canonical camera space; a per-image scale factor is carried alongside rather than silently applied:

outputs = model(pixel_values=pixel_values)
# dedicated post-process: crop preprocessing padding, resize back to (H, W)
results = processor.post_process_pointmap_estimation(outputs, target_sizes=[(H, W)] * N)
pts = np.stack([r["pointmap"].permute(1, 2, 0) for r in results])  # (N, H, W, 3) canonical
scales = outputs.scales                          # one scalar per image; XYZ / scale = metric

Each model_id is a lazy singleton in a registry, frozen and moved to the device once; inference is single-flight per engine.

4. Pointmap envelope

The output is a TEXT JSON envelope — the pointmap as a raw C-contiguous float32 buffer, base64-encoded (byte-exact across the server-mode HTTP boundary), plus one scale per image:

{"shape": [1, 480, 640, 3], "dtype": "float32", "b64": "<buffer>", "model_id": "facebook/sapiens2-pointmap-0.4b", "scales": [0.247]}

pointmap[n, y, x] is an (X, Y, Z) coordinate in canonical camera space, at the original input resolution. Divide the XYZ by scales[n] to convert to metric coordinates — the raw prediction is scale-canonical, so the scale factor travels with the buffer rather than being baked in.

5. Canvas Node

model_pointmap__estimate_pointmap

6. Server Mode

parallelism = "shared" (a stateless pointmap estimator — one server across eval workers); server_python = conda_env_python("ac-fm", "POINTMAP_PYTHON"). Pointmap estimation runs in the shared ac-fm env — Sapiens2ForPointmapEstimation is native to the transformers stack there. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_pointmap/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, a missing/malformed frame, or mixed input resolutions yields "" on the node's output. An empty envelope is a clean "no pointmap this step" signal; consumers keep their own fallback and never receive a fabricated 3D map.