Depth Anything NodeSet

The Depth Anything NodeSet (DepthAnythingNodeSet, workspace/nodesets/model/model_depth_anything.py) turns a single RGB frame into a dense depth map via Depth Anything V2 — the geometric-perception primitive for RGB-only agents. Obstacle avoidance, point-goal, sim2real, and lifting a 2D detection into 3D all need per-pixel depth without a depth sensor; this is the foundation-model way to get it.

env: ac-fm · backend: transformers AutoModelForDepthEstimation + AutoImageProcessor · default model_id: depth-anything/Depth-Anything-V2-Small-hf (relative)

1. Monocular depth — relative vs metric

The checkpoint decides the units, and the envelope's depth_type field labels them for the consumer:

If depth_type is left blank it is inferred from the model_id (metric when the id contains "metric", else relative) — so swapping to a metric checkpoint labels the output correctly with no extra config.

2. Forward & resolution

The processor resizes to the model's patch grid; one batched no-grad forward yields a low-resolution prediction, which is bicubic-upsampled back to the original input resolution:

pred = model(pixel_values=pixel_values).predicted_depth        # (N, h', w') at patch grid
pred = F.interpolate(pred.unsqueeze(1), size=(H, W),           # bicubic back to input size
                     mode="bicubic", align_corners=False).squeeze(1)   # (N, H, W)

All images in one call must share resolution — the envelope is a single (N, H, W) buffer. Mixed sizes degrade to empty with a self-log (split into uniform batches). In an agent loop N is typically 1 (per-frame depth).

3. Depth envelope

The output is a TEXT JSON envelope carrying the raw C-contiguous float32 buffer base64-encoded, plus model_id + depth_type self-description:

{"shape": [N, H, W], "dtype": "float32", "b64": "<base64 of the C-contiguous float32 buffer>", "model_id": "depth-anything/Depth-Anything-V2-Small-hf", "depth_type": "relative"}

Byte-exact across the server-mode HTTP boundary and ~4× smaller than a JSON float list.

4. Canvas Nodes

model_depth_anything__estimate_depth

Engines are lazy singletons in a registry keyed by model_id — relative and metric checkpoints coexist in one server. GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM under concurrent eval workers).

5. Server Mode

parallelism = "shared" (stateless estimator — one server across eval workers); server_python = conda_env_python("ac-fm", "DEPTH_ANYTHING_PYTHON"). AutoModelForDepthEstimation + the Depth-Anything-V2 checkpoints (ungated) are native to the shared ac-fm env (transformers 5.13), so no dedicated env is needed. The nodeset file stays Python-3.8-parseable (an override may point at a py3.8 env). Load: POST /api/components/nodesets/model_depth_anything/load?mode=server.

6. Environment Variables

7. Degraded Mode

A model/processor load failure latches (_load_failed — no retry storm); that, an images entry missing its base64, or a batch of mixed resolutions all yield depth = "" with a degraded self-log. An empty envelope is a clean "no depth this step" signal, never a fabricated map.