VGGT-SLAM 2 NodeSet — AgentCanvas

The VGGT-SLAM 2 NodeSet (ModelVggtSlam2NodeSet, workspace/nodesets/model/model_vggt_slam2/) wraps VGGT-SLAM 2.0 (MIT-SPARK, BSD-2-Clause; paper 26.01, RSS 2026) as a dense RGB-only SLAM session. Where pySLAM tracks per-frame with classic features, VGGT-SLAM batches keyframes into submaps (16 new frames + 1 overlap), runs a single VGGT feed-forward pass per submap, checks DINOv2-SALAD retrieval for loop closures, and aligns everything on the SL(4) projective manifold with GTSAM — absorbing the reconstruction ambiguity of unknown intrinsics instead of pretending it is a similarity transform. No depth input, no intrinsics input: VGGT self-calibrates.

Seven nodes: the streaming session (reset / track / finalize), the read-out surface (get_trajectory / get_map), the upstream eval ruler (eval_trajectory), and the optional open-set 3D object query (query_object — Perception Encoder CLIP retrieval + SAM 3 segmentation + PCA world-frame OBBs, upstream's --run_os path).

env: ac-vggt-slam conda (Python 3.11, torch 2.3.1+cu121, pip gtsam-develop with the SL(4) types) — server-mode, not a container · upstream pinned at MIT-SPARK/VGGT-SLAM @ 35327ac, pip-installed verbatim · parallelism="replicated" (the map is per-episode mutable state) · exclusive-GPU profiles (VGGT-1B bf16 ~5 GB weights + submap activations + SALAD).

VGGT-SLAM on TUM fr1/desk, RGB only — no depth, no intrinsics. The vggt_slam2_tum graph streams frames into the live session: the trajectory arrives in submap bursts (nothing, then ~17 keyframes at once, retro-corrected as optimization refines), and the dense map materializes chunk-by-chunk as each submap's VGGT forward pass lands (the in-canvas panels draw a 500k-point subsample). The closing orbit is a full-resolution render of the exported map artifact itself — all ~9.3M points of get_map's .npz handle.

Primitive	Purpose
`model_vggt_slam2__reset`	(Re)start the session: fresh Solver / map / SL(4) graph / SALAD database. VGGT-1B loads once per worker and is reused across episodes. `run_os=true` here if `query_object` will be used — CLIP embeddings are computed during tracking and cannot be backfilled.
`model_vggt_slam2__track`	Feed one RGB frame. Optical-flow keyframe gate → keyframe buffer → at `submap_size+1` frames: VGGT forward + loop-closure check + optimize. `pose` updates in submap bursts (see §2).
`model_vggt_slam2__finalize`	Flush the trailing partial submap — the streaming equivalent of upstream's last-image trigger. Fire once on `iterOut.final_stop`; its `ok` gates the read-out. Idempotent.
`model_vggt_slam2__get_trajectory`	Optimized keyframe trajectory: TUM-format text (upstream's exact pose writer) + parsed 4×4 poses + keyframe-aligned ground-truth rows captured by `track`.
`model_vggt_slam2__get_map`	Merged confidence-filtered world-frame cloud: upstream's `.pcd` writer verbatim (`pcd_path`) + the same cloud as an `.npz` handle (`map_handle` — what `pointCloudViewer` loads). Heavy geometry rides disk handles. `stream_on_submap=on` turns it into a live in-loop streamer: it exports only when a new submap has landed and returns a cheap empty handle otherwise — the demo's live map panel.
`model_vggt_slam2__eval_trajectory`	ATE with the upstream ruler: `evo_ape tum <gt> <est> -as` (Sim(3) alignment is mandatory — SL(4) poses are up-to-scale). Ground truth from the sequence name or the captured gt rows.
`model_vggt_slam2__query_object`	Open-set query over the built map: PE-CLIP text embedding → best keyframe (cosine over stored per-frame embeddings) → SAM 3 masks → world-frame points + PCA OBB per instance. Post-hoc; SAM 3 lazy-loads on first query.

1. Upstream anatomy & the driver mirror

Upstream ships the algorithm as a library (vggt_slam package: Solver, GraphMap, PoseGraph, ImageRetrieval, FrameTracker) but keeps the driver loop outside the package — main.py owns keyframe gating, submap triggering, and the optimize cadence, and setup.py does not ship it. The port therefore pip-installs the package verbatim (pinned commit) and mirrors the driver inside the nodeset backend (_backend.py), with a main.py:NN citation on every mirrored line:

Keyframe gate — Lucas–Kanade optical flow over goodFeaturesToTrack corners; a frame becomes a keyframe when mean displacement from the last keyframe exceeds min_disparity (frame_overlap.py, main.py:103). The first frame is always a keyframe.
Submap trigger — at submap_size + 1 buffered keyframes (16 new + 1 overlap; main.py:111): run_predictions (VGGT forward + SALAD retrieval + optional loop-closure re-inference with the fork's image_match_ratio gate) → add_points (submap + SL(4) factors, pairwise scale estimate) → graph.optimize() (Levenberg–Marquardt) → keep the last frame as the next submap's overlap (main.py:115-132).
Tail flush — upstream triggers processing on the last image regardless of buffer fill, including an overlap-only single-frame buffer — unless that same final frame already fired the size trigger. The port reproduces both branches exactly with a _last_call_processed flag consulted by finalize.
Invariants preserved — overlapping_window_size=1 and max_loops∈{0,1} only (upstream-supported values); poses recovered by RQ camera decomposition from optimized homographies; the VGGT model is injected per call, never owned by the Solver.

Upstream driver loop (top) and the port (bottom), column-aligned. The gate, buffer, and boundary processing collapse into track; the last-image flush becomes finalize on the loop's final side.

2. Session semantics: burst poses, timestamp-named keyframes

Pose lag is intrinsic, not a bug. track.pose is null until the first submap completes (~17 keyframes), then updates once per submap boundary — and earlier poses are retro-corrected by later optimization and loop closures. The authoritative trajectory is always get_trajectory fired after finalize. Graph authors wiring track.pose into a live viewer get the burst behaviour by design.

Keyframes hit disk, named by timestamp. Upstream's run_predictions consumes image file paths (its own streaming mode saves keyframes to disk too), so track persists each accepted keyframe as a lossless PNG named <timestamp>.png. The numeric stem is load-bearing: Submap.set_frame_ids parses it into the TUM pose-file timestamp column that evo associates against ground truth. Wire observe.timestamp into track.timestamp for eval-grade output.

One server process per eval worker. The accumulating map is per-episode state on the nodeset instance, so parallelism="replicated"; inside each process a 1-worker executor serializes all session calls (CUDA-context and GTSAM thread affinity). VGGT-1B, SALAD, and (under run_os) PE-CLIP/SAM 3 load once per process and survive episode resets.

3. Fidelity: deviations from upstream

Upstream is pip-installed verbatim at the pinned commit; the deviation set is enumerable and lives in the _backend.py docstring. Verdict: faithful with justified deviations (audited 2026-07-14/15; evidence in §4 — fr1_xyz reproduces the locally-run upstream ATE bit-exactly).

ID	Bucket	Deviation	Why · effect
D1	B — substrate-forced	`vggt_slam.solver.Viewer` swapped for a no-op stub at import time	Upstream's `Solver.__init__` unconditionally opens a viser server on port 8080; a headless multi-worker server cannot own a fixed port. No mainline math touches the viewer. Audit surface: `grep "self.viewer" solver.py`.
D2	C — cost-forced	SAM 3 lazy-loads on first `query_object` (upstream builds it eagerly under `--run_os`)	VRAM prudence only; the query math is identical.
D3	B — substrate-forced	Keyframes re-encoded to timestamp-named PNGs	A streaming source has no files; mirrors upstream's own `save_keyframe` path. Pixel-identical; filename drives evo association.
D4	B — substrate-forced	SAM 3 inference wrapped in `torch.autocast("cuda", bf16)`	The pinned sam3 on torch 2.3.1 dies with a BFloat16/Float matmul clash when called bare (as upstream's query loop does); autocast is SAM 3's documented usage. Verified standalone; no semantic effect.
—	C — env-forced (graph level)	`env_tum` replays the rgb/depth/gt associated frame stream; upstream globs the raw `rgb/` folder	fr1_desk: 613 raw frames → 596 associated (head-heavy drop) → keyframe sets diverge in the first second, then lock step. Measured effect: +0.6 mm ATE on desk, zero on xyz.

4. Verification & TUM parity

Upstream's README publishes no TUM numbers, so the reference was banked by running upstream main.py locally (eval_tum.sh's exact flags, same GPU) — twice, confirming it is run-to-run deterministic (0.024690 both runs on fr1_desk). All ATE numbers are evo_ape tum -as (Sim(3), scale-corrected — mandatory for SL(4) up-to-scale poses).

Check	Run	Result
Backend smoke (standalone, 200 fr1_desk frames)	in-env script	21 keyframes → 2 submaps (incl. tail flush), 22 poses, 3.3 M-point .pcd, ATE 0.0229
Graph smoke (scheduler, 200 frames)	`20260714_172702`	episode completed; metrics harvested (ate_rmse 0.023098, poses/submaps/loops)
Parity — fr1_xyz (full 798 frames)	`20260714_174221`	ATE 0.019903 vs upstream 0.019903 — bit-exact
Parity — fr1_desk (full sequence)	`20260714_174221`	ATE 0.02531 vs upstream 0.02469 (+0.6 mm) — fully attributed to the env-association row in §3
Open-set query ("keyboard", fr1_desk map)	`20260714_175351`	2 instances; keyboard-shaped OBBs (~19–35 cm × ~2 cm); overlay + points artifacts written

Two engine-level findings from this port, both fixed on the way in: (a) AutoServerApp used to initialize a second nodeset instance in on_startup while the /call handlers closed over the manifest-time throwaway — any session-type server nodeset served every call from a never-initialized instance (fixed in auto_server_app.py; belt-and-braces, nodes resolve the module singleton first). (b) The eval verdict harvest only reads graphOut sinks named metrics (dict, numeric fields flattened) or numeric scalars — any other name is skipped and the episode is convicted "no verdict".

5. Usage

Build the env once: bash scripts/install/install_ac_vggt_slam.sh (creates ac-vggt-slam; pins five upstream repos; fetches dino_salad.ckpt — upstream's own setup.sh does not, yet the Solver constructor loads it unconditionally). Fetch sequences: bash workspace/nodesets/env/env_tum/download.sh rgbd_dataset_freiburg1_desk.

/experiment:run smoke_vggt_slam2_tum vggt_slam2_tum split=freiburg1 episode_count=1 worker_count=1
/experiment:run perf_vggt_slam2_tum  vggt_slam2_tum split=freiburg1 episode_count=2 worker_count=1
/experiment:run vggt-slam2-tum-os    vggt_slam2_tum_os split=freiburg1 episode_count=1 worker_count=1

Profiles are exclusive-GPU with vram_mb deliberately at 8000 (≤ the admission ceiling — a declared marginal above usable_vram_mb can never admit and parks forever); exclusivity is the real guard. Graphs: workspace/graphs/vln/unverified/vggt_slam2_tum.json (+ _os variant); the same graph doubles as the GUI demo — open it on the canvas and press Play (the env panel defaults to the first sequence on disk), which is exactly what the clip above shows. Override the interpreter with $VGGT_SLAM_PYTHON.

6. What this nodeset is NOT

Not metric-scale SLAM. Poses live on SL(4) (up-to-scale, projective); evaluate with Sim(3) alignment, never raw. Downstream consumers needing metric scale must supply it.
Not per-frame SLAM. No pose until the first submap; ~17-keyframe latency between pose updates. Use pySLAM for per-frame tracking.
Not an env nodeset. No MDP, no episodes, no actions — feed frames in, read products out (model/ classification, same reasoning as pySLAM).
Not a general VGGT server. Single-shot N-view reconstruction lives in model_vggt; this nodeset is the streaming SLAM system built on the MIT-SPARK fork of VGGT.

7. Sources

VGGT-SLAM 2.0: Real-time Dense Feed-forward Scene Reconstruction (26.01, RSS 2026) · VGGT-SLAM 1.0 (NeurIPS 2025) — Maggio & Carlone, MIT-SPARK.
Upstream: MIT-SPARK/VGGT-SLAM @ 35327ac (BSD-2-Clause); VGGT fork VGGT_SPARK @ 6e6e161; salad @ 33ca9c0; perception_models @ 3e352cc; sam3 @ 5dd401d.
Port: workspace/nodesets/model/model_vggt_slam2/ (__init__.py nodes; _backend.py driver mirror with main.py:NN citations + deviation list D1–D4); env recipe scripts/install/install_ac_vggt_slam.sh + scripts/install/envs/ac_vggt_slam.lock.