VGGT NodeSet

The VGGT NodeSet (VggtNodeSet, workspace/nodesets/model/model_vggt.py) wraps VGGT (Visual Geometry Grounded Transformer, CVPR 2025) as a feed-forward 3D reconstruction primitive. Where Depth Anything gives per-view relative depth, VGGT ingests 1→N RGB views and, in a single forward pass, regresses camera poses, dense per-view depth, and a dense world-frame point map — no SLAM, no pairwise global-alignment loop. That turns a bare RGB stream into geometry, the substrate open-vocabulary 3D mapping (VGGT points + CLIP features) and RGB-only navigation build on.

env: ac-vggt (dedicated) · backend: standalone vggt package (VGGT.from_pretrained via huggingface_hub) · default model_id: facebook/VGGT-1B (cc-by-nc-4.0; -Commercial variant for commercial use)

1. One forward, three heads

reconstruct runs VGGT's camera, depth, and point heads together in a single model(images) call — that is the model's native shape, so splitting it into per-head nodes would only re-run the shared transformer. The three output ports carry the three head groups; a graph wires whichever it needs (depth alone for obstacle checks, cameras + points for mapping). Arrays are ordered by input view (leading dim S = number of views).

track_points is split out as a second node precisely because it is the only surface that needs an extra input — the query_points that drive the track head. Everything else in VGGT is view-only.

2. Preprocessing verbatim, then pose decode

Images are preprocessed by VGGT's own load_and_preprocess_images — reused verbatim, with zero reimplementation. PIL's Image.open accepts a file-like object, so decoded base64 is handed in as BytesIO rather than re-deriving the resize / divisible-by-14 / crop-vs-pad logic (which would risk drift from upstream). The camera head emits a compact 9-D pose encoding; it is decoded to explicit matrices via pose_encoding_to_extri_intri — extrinsics [S,3,4] (OpenCV, cam-from-world) and intrinsics [S,3,3] (principal point assumed at the image centre, VGGT's own convention):

# One forward runs the camera + depth + point heads together (VGGT's native shape).
images = load_and_preprocess_images(files, mode=mode).to(device)  # BytesIO fed verbatim
with torch.no_grad(), self._autocast():
    pred = model(images)                       # pose_enc, depth(+conf), world_points(+conf)
# 9-D pose encoding -> extrinsic[S,3,4] + intrinsic[S,3,3] (centred principal point)
extr, intr = pose_encoding_to_extri_intri(pred["pose_enc"], (H, W))

The aggregator runs under a bf16/fp16 autocast (the upstream recipe); the camera/depth/point heads re-disable it internally to fp32. The track head, however, runs inside the outer autocast, so its outputs are cast back to float32 before leaving the GPU (numpy has no bfloat16).

3. World points — head vs unproject

VGGT gives two routes to a world-frame point cloud, selected by the points_source config:

• head (default) — the point head's direct world_points output.
• unproject — back-projecting the predicted depth through the predicted camera (unproject_depth_map_to_point_map). The upstream README notes this depth × camera path is often the sharper point cloud.

Either way the world_points port carries a (S, H, W, 3) array plus its world_points_conf (S, H, W); the envelope records which source produced it.

4. Multi-array envelope

Each port is a TEXT JSON envelope. Unlike the single-matrix embedding envelope (CLIP / DINOv2), a VGGT port carries several named arrays — each the raw C-contiguous float32 buffer base64-encoded (byte-exact across the server-mode HTTP boundary, ~4× smaller than a JSON float list) — alongside model_id and the preprocessed image_hw:

{"model_id": "facebook/VGGT-1B", "image_hw": [392, 518],
 "extrinsics": {"shape": [3, 3, 4], "dtype": "float32", "b64": "<C-contiguous float32 buffer>"},
 "intrinsics": {"shape": [3, 3, 3], "dtype": "float32", "b64": "…"}}

Consumers read env["extrinsics"]["b64"] etc. and np.frombuffer(...).reshape(shape). The depth port carries depth (S,H,W) + depth_conf (S,H,W); world_points carries world_points (S,H,W,3) + world_points_conf (S,H,W) + points_source; tracks carries track (S,N,2) + vis (S,N) + conf (S,N).

5. Canvas Nodes

model_vggt__reconstruct

model_vggt__track_points

Note that query_points are in the preprocessed image frame (width 518 in crop mode), not the original resolution — the caller reads image_hw from a prior reconstruct (or the same input) to place them. Engines are lazy singletons in a registry keyed by model_id; GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM — VGGT's point/depth maps scale with view count × resolution).

6. Server Mode

parallelism = "shared" (stateless geometry primitives — one server across eval workers); server_python = conda_env_python("ac-vggt", "VGGT_PYTHON"). VGGT is a dedicated env, not a piggyback on the shared ac-fm: it is the standalone vggt package (loaded via huggingface_hub, not transformers) and it pins numpy<2, which cannot coexist with ac-fm's numpy-2 stack without downgrading numpy under the other FM nodesets. The file stays Python-3.8-parseable (an override may point at a py3.8 env). Load: POST /api/components/nodesets/model_vggt/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, an images entry missing its base64, or (for tracking) absent/malformed query_points, yields "" on every output of that node. Consumers keep their own fallback — an empty envelope is a clean "no reconstruction this step" signal, never fabricated geometry.