DepthPro NodeSet

The DepthPro NodeSet (DepthProNodeSet, workspace/nodesets/model/model_depthpro.py) wraps Apple DepthPro for zero-shot metric depth: one RGB frame in, a dense depth map in metres plus the recovered field of view out. It is the absolute-scale companion to Depth Anything — where Depth Anything gives relative / inverse depth, DepthPro gives distance you can act on in metres, and recovers focal length from a lone RGB frame. Obstacle distance, metric 3D lifting, and sim2real scale all want this.

env: ac-fm (shared FM env) · backend: transformers AutoModelForDepthEstimation + AutoImageProcessor · default: apple/DepthPro-hf (ungated)

1. One primitive: RGB → metric depth

The tool takes a list of same-resolution images and returns, per image, a per-pixel depth in metres plus the horizontal field of view in degrees (DepthPro estimates focal length as part of the forward). It shares the Depth Anything node's input contract and envelope convention, so the two compose as a relative/metric depth pair — reach for DepthPro when the downstream needs true scale.

2. Images in — uniform resolution

The images port accepts a list of {rgb_base64} dicts (or raw base64 strings). All images in one call must share resolution — the envelope is a single (N, H, W) buffer, so mixed sizes degrade to empty with a self-log. In an agent loop N is typically 1 (per-frame depth).

3. Model & cost

The engine loads AutoModelForDepthEstimation + AutoImageProcessor for the configured model_id. Post-processing recovers metric depth and FoV and resizes depth back to the original resolution:

outputs = model(pixel_values=pixel_values)
results = processor.post_process_depth_estimation(outputs, target_sizes=target_sizes)
depth = results[0]["predicted_depth"]   # (H, W) metres
fov   = results[0]["field_of_view"]    # degrees

Cost note: DepthPro runs at a fixed high internal resolution (1536²) regardless of input size, so a single forward is heavier than Depth Anything's — treat it as the "metric when you need it" companion, not a per-step replacement. Each model_id is a lazy singleton, frozen and moved to the device once; inference is single-flight per engine.

4. Depth envelope

The output is a TEXT JSON envelope — the depth as a raw C-contiguous float32 buffer, base64-encoded (byte-exact across the server-mode HTTP boundary, ~4× smaller than a JSON float list):

{"shape": [1, 480, 640], "dtype": "float32", "b64": "<buffer>",
 "model_id": "apple/DepthPro-hf", "depth_type": "metric", "field_of_view": [58.7]}

Each pixel is depth in metres at the original input resolution; depth_type is always "metric" (the whole point of this node vs relative depth), and field_of_view gives the horizontal FoV in degrees per image.

5. Canvas Node

model_depthpro__estimate_metric_depth

6. Server Mode

parallelism = "shared" (a stateless depth estimator — one server across eval workers); server_python = conda_env_python("ac-fm", "DEPTHPRO_PYTHON"). DepthPro runs in the shared ac-fm env — it is native to the transformers stack there. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_depthpro/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, a missing/malformed frame, or mixed input resolutions yields "" on the node's output. An empty envelope is a clean "no depth this step" signal; consumers keep their own fallback and never receive a fabricated depth map.