pySLAM NodeSet

The pySLAM NodeSet (ModelPySlamNodeSet, workspace/nodesets/model/model_pyslam/) wraps pySLAM (Luigi Freda, GPL-3.0) as a streaming visual-SLAM session. Where VGGT reconstructs geometry from N views in one feed-forward pass, pySLAM ingests an RGB-D stream one frame at a time and incrementally tracks the camera pose while accumulating a sparse 3-D map — classic feature SLAM (ORB features + DBoW3 loop closure, C++ core), a moving camera building a map as it goes. Feed it a policy's egocentric frames and it reconstructs the walked scene alongside the run.

The nodeset has eleven nodes in three groups: the streaming session above (five nodes — the four core verbs plus a dense volumetric map); three stateless perception nodes that use pySLAM's standalone classes as pure functions — feature extraction, feature matching, and evo trajectory evaluation (ATE / RPE); and three neural full-surface nodes — dense depth prediction, semantic segmentation, and feed-forward multi-view reconstruction (DUSt3R / MASt3R / VGGT). The last two groups share the same container but never touch the map. The aim is to expose pySLAM's whole capability surface — AgentCanvas supporting pySLAM-the-model, not a curated subset for one downstream graph.

env: container — agentcanvas/pyslam:cpu-fixed for the SLAM / feature / depth / semantic path, agentcanvas/pyslam:cuda (adds compiled multi-view backends) for reconstruct_multiview, rootless Docker, not a conda env · backend: GPL-3.0 pySLAM behind an HTTP bridge, never imported in the framework process · front-end: ORB2 features + DBOW3 loop closure, RGB-D by default · GPU via rootless CDI, on by default

1. The container bridge

pySLAM is GPL-3.0 and carries a native C++ core (g2o, Pangolin, DBoW3) pinned to Python 3.11.9. Rather than vendor that source or build a sibling conda env, the nodeset keeps the whole dependency behind a container boundary — the same treatment habitat_sim gets. Three layers sit between a canvas node and the SLAM math; the framework side never runs import pyslam, and every per-frame call is one HTTP round-trip to a shim running inside the container.

The nodeset directory is bind-mounted read-only into the container, so _server.py/_backend.py run inside without baking our code into the image — only fastapi/uvicorn and pySLAM live there. Container access is forced onto the user's rootless Docker daemon (DOCKER_HOST/XDG_RUNTIME_DIR set to the user socket), so the whole thing needs no sudo.

One call, end to end. The graph executor fires a freshly-constructed node each step, so a node cannot hold the session on self — it reaches the shared client through the process-level singleton _NODESET that initialize() set (§3). A single track then flows node → client → shim → session: the client serialises the frame and POSTs it over the loopback port; the shim hands it to the PySlamSession, which marshals the call onto its one worker thread and returns the pose as JSON. The single thread is not incidental — pySLAM's local-mapping / loop-closing / volumetric background threads are bound to the thread that constructed Slam, so every track must run on that same thread (the same single-thread-affinity discipline the GL/physics env nodesets use).

What rides the wire. Two payload classes cross the bridge differently. Light results (pose, tracking state, point counts) travel as JSON; heavy geometry — frames in, sparse / dense maps and multi-view clouds out — travels as raw .npz bytes over an octet-stream, never base64-in-JSON. And on the graph wire only a handle path travels: the client writes the returned .npz to a host-side file (framework-user-owned, sidestepping rootless-Docker uid remapping) and passes that path downstream, so a multi-thousand-point map never inflates the executor's in-memory state.

The session's shape is fixed at launch, not per call. Sensor mode, feature / loop presets, volumetric integration, indoor/outdoor regime — all chosen when the container starts: initialize() passes them as -e environment variables on docker run, and the shim reads them when it lazily builds the session (_get_session()). That is why those knobs live as environment variables (§7) rather than call arguments — they define the container, while only per-frame data (images, depth) rides the HTTP body. The per-node Reset config fields override the env defaults right before the first configure_camera.

2. A streaming session, not a stateless pass

Every other foundation-model nodeset here is a stateless primitive (parallelism = "shared", one server across eval workers). pySLAM is the exception: its map is mutable per-episode state that accumulates across track calls, so it is a session. Consequences:

• parallelism = "replicated" — each batch-eval worker needs its own map, hence its own container (one container per client).
• server_python = None — the nodeset runs in local mode: it is not itself a server-mode subprocess (the container is). It reaches pySLAM over HTTP, not by importing it.
• The Slam instance is built lazily on the first reset (once camera intrinsics are known), then reused; per-episode reset only clears the map.

The three Tier-2 perception nodes (§5) are the counterexample within this nodeset: they are stateless pure functions that reuse the same container but never touch the Slam object, so — unlike track / get_map — they do not gate on the session being built.

3. Weaving a passive observer into the dataflow

A SLAM observer riding alongside a navigation policy is easy to wire wrong, because two graph-executor facts fight the naïve topology:

• Fresh instances. The executor fires freshly-constructed node_cls() objects, not the get_tools(self) instances that carry a nodeset back-reference. So nodes reach the shared session through a process-level singleton that initialize() sets, read as self._nodeset or _NODESET — safe because local mode is always single-process.
• Demand-driven pruning. A node whose outputs feed nothing is pruned and never fires. A passive observer therefore must be woven into the live path: track's num_map_points is wired to an iterOut port, and reset's episode_ok to an iterIn init port, so both actually run every step / once per episode.

The shipped example, vln/unverified/pyslam_slam_probe.json, is a pure-policy VLN-CE navigator (CMA policy, no LLM) with pySLAM observing: the policy drives, pySLAM reconstructs the walked scene, and the wiring above keeps the observer alive.

4. Camera intrinsics & depth scale

Two alignment details make an environment's RGB-D stream usable by pySLAM:

• Intrinsics. Wire the environment's intrinsics output (Habitat's observe port emits {fx, fy, cx, cy, width, height}) into reset.intrinsics for an exact pinhole — no hard-coded HFOV. Absent that, cam_hfov + width/height derive fx from the field of view.
• Depth scale. RGB-D SLAM needs depth in metres. Habitat VLN-CE renders depth normalised to [0,1] over a 10 m range, so set depth_scale = 10.0 on the reset node — the track node multiplies depth back to metric before feeding pySLAM. (pySLAM's own DepthMapFactor applies only to file-loaded depth, not an ndarray passed to track(), so the scaling is node-side.) When RGB and depth grids differ (e.g. CMA's RGB 224 vs depth 256), depth is nearest-neighbour resampled onto the RGB grid first.

5. Canvas Nodes

model_pyslam__reset

model_pyslam__track

{
  "pose": [[1.0, 0.0, 0.0, 0.12],
           [0.0, 1.0, 0.0, 0.03],
           [0.0, 0.0, 1.0, 0.58],
           [0.0, 0.0, 0.0, 1.0]],
  "tracking_state": "OK",          
  "num_map_points": 1862
}

tracking_state walks NOT_INITIALIZED → OK (→ LOST if track drops).

model_pyslam__get_trajectory

model_pyslam__get_map

{
  "map_handle": "outputs/pyslam_maps/map_0001_001862.npz",
  "num_points": 1862,
  "num_keyframes": 12
}

The .npz holds a points array of shape (N, 3) — the sparse landmark cloud in world coordinates.

model_pyslam__get_dense_map

Export the dense map — not the sparse landmark cloud but pySLAM's own volumetric integrator output, a fused voxel point cloud or TSDF mesh built over the tracked keyframes. A session verb: it requires the session to be built with volumetric integration on (PYSLAM_VOLUMETRIC=1) and returns an empty handle otherwise.

The drain-wait matters: the volumetric integrator consumes keyframes on its own process via multiprocessing queues, so popping an output before the queue drains returns a partial or empty map. The node blocks until the queue is empty, then extracts a complete volume (VOXEL_GRID is the verified default).

The first five nodes are the streaming session; the next three are stateless perception — they reuse the container but hold no map state, so they never gate on the session being built.

model_pyslam__extract_features

pySLAM's local-feature front-end as a standalone tool — detect keypoints and compute descriptors on one image.

ORB2 (the ORB-SLAM2 interface) needs a C++ binding the CPU image lacks — use plain ORB; learned detectors auto-download a checkpoint on first use.

model_pyslam__match_features

Match two descriptor blocks (from Extract Features) → matched index pairs. Classical matchers run on the arrays alone — no images, no weights.

model_pyslam__eval_trajectory

Score an estimated trajectory against ground truth — ATE + RPE via evo. Pairs naturally with Get Trajectory. Pure math, no weights, no GPU.

A near-straight-line trajectory (a straight-corridor episode) makes Umeyama alignment rank-deficient, so ate_rmse can come back null (evo cannot align) while rpe_rmse still returns via a no-align fallback — an evo limitation, not a fault.

The last three nodes are neural full-surface — pySLAM ships a bank of learned dense-perception backends behind one factory each, and exposing pySLAM's whole surface means these get first-class nodes too. Like the stateless perception nodes they hold no map state, but they run neural nets on the GPU, so the first call loads (or downloads) a checkpoint. (The streaming session's own algorithm variety — feature detector, loop detector, sensor mode — is reached by config passthrough on the Reset node, not by new nodes.)

model_pyslam__predict_depth

Dense metric depth from a single image — pySLAM's depth-estimator bank as a standalone tool.

Mono estimators use rgb alone; stereo ones also consume image_right. SGBM is pure OpenCV/CPU; the learned estimators run on the GPU and auto-download a checkpoint on first use.

model_pyslam__segment_semantic

Per-pixel semantic segmentation — pySLAM's segmentation bank as a standalone tool.

model_pyslam__reconstruct_multiview

Feed-forward multi-view 3-D reconstruction — pySLAM's scene-from-views stack (the DUSt3R / MASt3R / VGGT family). Give a list of ≥2 overlapping views; get a fused global point cloud plus per-view camera poses in one pass. Needs the agentcanvas/pyslam:cuda image (the multi-view backends carry compiled CUDA kernels).

Weights are external, not baked into the image (§6): MASt3R / MV-DUSt3R load checkpoints from a mounted data/models/pyslam/ folder; the HuggingFace-runtime backends (VGGT / VGGT-Robust / Depth-Anything V3 / Fast3r / pure DUSt3R) download into a mounted cache on first use. MV-DUSt3R is a known gap — a version drift in pySLAM's bundled copy leaves it unusable; the other backends cover multi-view. MASt3R is the verified default (a 104k-point Chateau reconstruction runs end-to-end).

6. Deployment & lifecycle

initialize() constructs the client and docker run -ds the image (nodeset dir bind-mounted at /opt/bridge, a free loopback port published), then polls /health until the shim answers — the blocking Docker call offloaded off the event loop. shutdown() posts /close (quits Slam, joins its background threads) then docker rm -fs the container. The image is built once via rootless Docker; pySLAM is cloned and compiled at build time, never entering the repo.

GPU. The container is launched with --device nvidia.com/gpu=all so the neural backends (learned depth, semantic segmentation, multi-view reconstruction) run on the GPU — the classic SLAM core stays on CPU. Rootless Docker reaches the GPU through CDI (the legacy --gpus cgroup path fails unprivileged). GPU is on by default and falls back to CPU when unavailable — set PYSLAM_GPU=0 to force CPU.

Two images, auto-selected. cpu-fixed already carries a CUDA torch ("cpu" only means the SLAM C++ core is CPU-built), so the depth and semantic backends run on the GPU there with no separate image. The multi-view backends are the exception: they carry compiled CUDA kernels (curope) that must be built with nvcc present, so reconstruct_multiview needs the heavier agentcanvas/pyslam:cuda image (built by docker/Dockerfile.cuda). When PYSLAM_IMAGE is unset the client picks the image by GPU: GPU present → :cuda (the whole surface works out of the box), no GPU → :cpu-fixed (the leaner image; multi-view needs a GPU anyway). A GPU start that fails on a host without CDI also degrades to the CPU image. Pin PYSLAM_IMAGE to override.

Weights are external. Neither image bakes in model weights — they carry only code and compiled kernels. The multi-view checkpoints live in a host data/models/pyslam/ folder and mount read-only into the container at runtime (PYSLAM_WEIGHTS_DIR overrides the location); the HuggingFace-runtime backends download into a mounted cache on first use. Fetch the explicit checkpoints with the tracked scripts under docker/weights/:

cd workspace/nodesets/model/model_pyslam/docker/weights
bash download_mast3r.sh      # MASt3R (~2.5GB) — the verified default
bash download_all.sh         # mast3r + mvdust3r explicit checkpoints

export DOCKER_HOST="unix:///run/user/$(id -u)/docker.sock"
bash workspace/nodesets/model/model_pyslam/docker/build.sh agentcanvas/pyslam:cpu-fixed

Run the probe graph through the experiment wrapper. worker_count=1 is required — a higher count auto-routes the nodeset to server mode, which the container-bridge (local-mode) design does not support.

# One episode of the pure-policy probe graph, submitted to the slot backend.
# worker_count=1 is REQUIRED: >1 auto-routes the nodeset to server mode,
# which the container-bridge (local-mode) nodeset does not support.
AGENTCANVAS_BACKEND_URL=http://127.0.0.1:8002 \
  python .claude/commands/experiment/bin/submit.py \
  architect-cma pyslam_slam_probe \
  episode_count=1 worker_count=1 split=val_unseen step_budget=80

7. Environment Variables

Per-node config fields on the Reset node (sensor type, presets, HFOV, depth scale) override these env defaults.

8. Operational notes

• Bootstrap latency. RGB-D SLAM needs a few frames of parallax to initialise. On a validated Habitat run the front-end spent ~16 discrete steps in NOT_INITIALIZED before locking on, then tracked steadily — reconstructing a metric room-scale cloud (~1.5–1.9k points, core extent ≈ 5 × 3 × 7 m) matching the walked path.
• Coverage is episode-dependent. Episodes dominated by in-place rotation, very short paths, or low-texture views may never accumulate enough parallax to initialise, and legitimately return zero map points — an inherent property of feature SLAM under discrete large-rotation motion, not a fault. The front-end honestly reports "not initialised" rather than emitting spurious geometry.
• Per-episode reset is clean. Across a multi-episode run each episode re-bootstraps from an empty map (reset_session); maps never carry over, and a failed-to-initialise episode exports an empty cloud rather than the previous episode's points.