pySLAM NodeSet

The pySLAM NodeSet (ModelPySlamNodeSet, workspace/nodesets/model/model_pyslam/) wraps pySLAM (Luigi Freda, GPL-3.0) as a streaming visual-SLAM session. Where VGGT reconstructs geometry from N views in one feed-forward pass, pySLAM ingests an RGB-D stream one frame at a time and incrementally tracks the camera pose while accumulating a sparse 3-D map — classic feature SLAM (ORB features + DBoW3 loop closure, C++ core), a moving camera building a map as it goes. Feed it a policy's egocentric frames and it reconstructs the walked scene alongside the run.

The nodeset has eleven nodes in three groups: the streaming session above (five nodes — the four core verbs plus a dense volumetric map); three stateless perception nodes that use pySLAM's standalone classes as pure functions — feature extraction, feature matching, and evo trajectory evaluation (ATE / RPE); and three neural full-surface nodes — dense depth prediction, semantic segmentation, and feed-forward multi-view reconstruction (DUSt3R / MASt3R / VGGT). The last two groups share the same container but never touch the map. The aim is to expose pySLAM's whole capability surface — AgentCanvas supporting pySLAM-the-model, not a curated subset for one downstream graph.

env: Container Launch (ADR-server-005) — server_image = agentcanvas/pyslam:cuda by default (full surface incl. reconstruct_multiview; PYSLAM_IMAGE pins :cpu-fixed for the leaner build), rootless Docker, not a conda env · backend: GPL-3.0 pySLAM stays inside the container, never imported in the framework process · front-end: ORB2 features + DBOW3 loop closure, RGB-D by default · GPU via rootless CDI, on by default

1. Container Launch

pySLAM is GPL-3.0 and carries a native C++ core (g2o, Pangolin, DBoW3) pinned to Python 3.11.9. Rather than vendor that source or build a sibling conda env, the nodeset keeps the whole dependency behind a container boundary — the same treatment habitat_sim gets. Since 2026-08-20 the boundary is the framework's own Container Launch vehicle (design doc, ADR-server-005): the nodeset declares server_image and the registry routes it to server mode with docker run as the launch vehicle — the stock auto_host serves the manifest protocol from inside the container, and the framework talks to auto-generated proxy nodes. The framework side never runs import pyslam; the whole nodeset module (nodes + PySlamSession) executes in-container. This replaced the nodeset's original private bridge (a hand-written _client.py docker driver + an in-container FastAPI shim), which was this mechanism's validated prototype.

No framework or nodeset code is baked into the image — the repo rides the read-only mount, so a framework upgrade never needs an image rebuild. The image carries only pySLAM's own venv (which already includes the requirements-serve.txt set). Container access is forced onto the user's rootless Docker daemon (DOCKER_HOST/XDG_RUNTIME_DIR set to the user socket), so the whole thing needs no sudo.

One call, end to end. On the canvas the pySLAM nodes are proxy nodes the registry generated from the container's /manifest. A single track flows proxy → auto_host → node → session: the proxy POSTs the inputs to /call/model_pyslam__track over the loopback port (numpy arrays ride msgpack natively — no base64, no custom wire protocol); auto_host dispatches to the real track node running in-container, which hands the frame to the PySlamSession, which marshals the call onto its one worker thread and returns the pose. The single thread is not incidental — pySLAM's local-mapping / loop-closing / volumetric background threads are bound to the thread that constructed Slam, so every track must run on that same thread (the same single-thread-affinity discipline the GL/physics env nodesets use).

What rides the wire. Light results (pose, tracking state, point counts) and frames travel on the manifest protocol like any server-mode nodeset's. Heavy geometry — sparse / dense maps and multi-view clouds — never rides a wire at all: the in-container node writes the .npz into /opt/out (the server_mounts rw drop, host side outputs/pyslam_maps/) and passes only the host-resolvable handle path downstream, so a multi-thousand-point map never inflates the executor's in-memory state. The private HTTP endpoint zoo of the old bridge (/track, /get_map npz-over-octet-stream, …) is gone — the stock GET /manifest · POST /call/{fn} · GET /health surface replaced all of it.

The session's shape is fixed at launch, not per call. Sensor mode, feature / loop presets, volumetric integration, indoor/outdoor regime — all chosen when the container starts: the class-level server_env snapshots every PYSLAM_* variable from the host environment at import time and the registry injects them as -e on docker run; the in-container initialize() reads them back when it builds the session. That is why those knobs live as environment variables (§7) rather than call arguments — they define the container, while only per-frame data (images, depth) rides the call. The per-node Reset config fields override the env defaults right before the first configure_camera.

2. A streaming session, not a stateless pass

Every other foundation-model nodeset here is a stateless primitive (parallelism = "shared", one server across eval workers). pySLAM is the exception: its map is mutable per-episode state that accumulates across track calls, so it is a session. Consequences:

• parallelism = "replicated" — each batch-eval worker needs its own map, hence its own container (server-mode fan-out spawns one ContainerServer per worker).
• server_image — the nodeset runs in server mode with the container as the launch vehicle: the whole module executes inside the container, and the framework only ever talks to its proxy nodes.
• The Slam instance is built lazily on the first reset (once camera intrinsics are known), then reused; per-episode reset only clears the map.

The three Tier-2 perception nodes (§5) are the counterexample within this nodeset: they are stateless pure functions that reuse the same container but never touch the Slam object, so — unlike track / get_map — they do not gate on the session being built.

3. Weaving a passive observer into the dataflow

A SLAM observer riding alongside a navigation policy is easy to wire wrong, because two graph-executor facts fight the naïve topology:

• Fresh instances. The host executor fires freshly-constructed node_cls() objects, not the get_tools(self) instances that carry a nodeset back-reference — and in-container the same discipline holds as a safety net. So nodes reach the shared session as self._nodeset or _NODESET, a process-level singleton that initialize() sets — safe because one container is one process (replicated: one per worker).
• Demand-driven pruning. A node whose outputs feed nothing is pruned and never fires. A passive observer therefore must be woven into the live path: track's num_map_points is wired to an iterOut port, and reset's episode_ok to an iterIn init port, so both actually run every step / once per episode.

The shipped example, vln/unverified/pyslam_slam_probe.json, is a pure-policy VLN-CE navigator (CMA policy, no LLM) with pySLAM observing: the policy drives, pySLAM reconstructs the walked scene, and the wiring above keeps the observer alive.

4. Camera intrinsics & depth scale

Two alignment details make an environment's RGB-D stream usable by pySLAM:

• Intrinsics. Wire the environment's intrinsics output (Habitat's observe port emits {fx, fy, cx, cy, width, height}) into reset.intrinsics for an exact pinhole — no hard-coded HFOV. Absent that, cam_hfov + width/height derive fx from the field of view.
• Depth scale. RGB-D SLAM needs depth in metres. Habitat VLN-CE renders depth normalised to [0,1] over a 10 m range, so set depth_scale = 10.0 on the reset node — the track node multiplies depth back to metric before feeding pySLAM. (pySLAM's own DepthMapFactor applies only to file-loaded depth, not an ndarray passed to track(), so the scaling is node-side.) When RGB and depth grids differ (e.g. CMA's RGB 224 vs depth 256), depth is nearest-neighbour resampled onto the RGB grid first.

5. Canvas Nodes

model_pyslam__reset

model_pyslam__track

{
  "pose": [[1.0, 0.0, 0.0, 0.12],
           [0.0, 1.0, 0.0, 0.03],
           [0.0, 0.0, 1.0, 0.58],
           [0.0, 0.0, 0.0, 1.0]],
  "tracking_state": "OK",          
  "num_map_points": 1862
}

tracking_state walks NOT_INITIALIZED → OK (→ LOST if track drops).

model_pyslam__get_trajectory

model_pyslam__get_map

{
  "map_handle": "<repo>/outputs/pyslam_maps/map_000060.npz",
  "num_points": 1862,
  "num_keyframes": 12
}

The .npz holds a points array of shape (N, 3) — the sparse landmark cloud in world coordinates.

model_pyslam__get_dense_map

Export the dense map — not the sparse landmark cloud but pySLAM's own volumetric integrator output, a fused voxel point cloud or TSDF mesh built over the tracked keyframes. A session verb: it requires the session to be built with volumetric integration on (PYSLAM_VOLUMETRIC=1) and returns an empty handle otherwise.

The drain-wait matters: the volumetric integrator consumes keyframes on its own process via multiprocessing queues, so popping an output before the queue drains returns a partial or empty map. The node blocks until the queue is empty, then extracts a complete volume (VOXEL_GRID is the verified default).

The first five nodes are the streaming session; the next three are stateless perception — they reuse the container but hold no map state, so they never gate on the session being built.

model_pyslam__extract_features

pySLAM's local-feature front-end as a standalone tool — detect keypoints and compute descriptors on one image.

ORB2 (the ORB-SLAM2 interface) needs a C++ binding the CPU image lacks — use plain ORB; learned detectors auto-download a checkpoint on first use.

model_pyslam__match_features

Match two descriptor blocks (from Extract Features) → matched index pairs. Classical matchers run on the arrays alone — no images, no weights.

model_pyslam__eval_trajectory

Score an estimated trajectory against ground truth — ATE + RPE via evo. Pairs naturally with Get Trajectory. Pure math, no weights, no GPU.

A near-straight-line trajectory (a straight-corridor episode) makes Umeyama alignment rank-deficient, so ate_rmse can come back null (evo cannot align) while rpe_rmse still returns via a no-align fallback — an evo limitation, not a fault.

The last three nodes are neural full-surface — pySLAM ships a bank of learned dense-perception backends behind one factory each, and exposing pySLAM's whole surface means these get first-class nodes too. Like the stateless perception nodes they hold no map state, but they run neural nets on the GPU, so the first call loads (or downloads) a checkpoint. (The streaming session's own algorithm variety — feature detector, loop detector, sensor mode — is reached by config passthrough on the Reset node, not by new nodes.)

model_pyslam__predict_depth

Dense metric depth from a single image — pySLAM's depth-estimator bank as a standalone tool.

Mono estimators use rgb alone; stereo ones also consume image_right. SGBM is pure OpenCV/CPU; the learned estimators run on the GPU and auto-download a checkpoint on first use.

model_pyslam__segment_semantic

Per-pixel semantic segmentation — pySLAM's segmentation bank as a standalone tool.

model_pyslam__reconstruct_multiview

Feed-forward multi-view 3-D reconstruction — pySLAM's scene-from-views stack (the DUSt3R / MASt3R / VGGT family). Give a list of ≥2 overlapping views; get a fused global point cloud plus per-view camera poses in one pass. Needs the agentcanvas/pyslam:cuda image (the multi-view backends carry compiled CUDA kernels).

Weights are external, not baked into the image (§6): MASt3R / MV-DUSt3R load checkpoints from a mounted data/models/pyslam/ folder; the HuggingFace-runtime backends (VGGT / VGGT-Robust / Depth-Anything V3 / Fast3r / pure DUSt3R) download into a mounted cache on first use. MV-DUSt3R is a known gap — a version drift in pySLAM's bundled copy leaves it unusable; the other backends cover multi-view. MASt3R is the verified default (a 104k-point Chateau reconstruction runs end-to-end).

6. Deployment & lifecycle

Loading the nodeset makes the registry's ContainerServer run a foreground docker run --rm --init (repo bind-mounted read-only at /opt/agentcanvas-workspace, weights and the /opt/out handle drop mounted per server_mounts, a loopback port published), then poll /health until the stock auto_host answers — auto_host has already called the nodeset's initialize() by then. Teardown is framework-owned too: auto_host runs shutdown() (quits Slam, joins its background threads), then ContainerServer issues docker stop and --rm reaps the container — a SIGKILLed backend can no longer orphan it. The image is built once via rootless Docker; pySLAM is cloned and compiled at build time, never entering the repo.

GPU. With server_image_gpu on (the default) the container is launched with --device nvidia.com/gpu=all so the neural backends (learned depth, semantic segmentation, multi-view reconstruction) run on the GPU — the classic SLAM core stays on CPU. Rootless Docker reaches the GPU through CDI (the legacy --gpus cgroup path fails unprivileged). When the GPU device is absent, ContainerServer retries once without it — set PYSLAM_GPU=0 to skip the GPU claim outright.

One default image. server_image defaults to agentcanvas/pyslam:cuda — the full surface out of the box. Its neural layers load lazily, so the same image runs fine CPU-only; the old bridge's per-GPU tag auto-selection is gone. cpu-fixed still exists as the leaner base (it already carries a CUDA torch — "cpu" only means the SLAM C++ core is CPU-built — but lacks the compiled multi-view kernels, so reconstruct_multiview needs :cuda). Pin PYSLAM_IMAGE=agentcanvas/pyslam:cpu-fixed to use it.

Identity alignment, not open perms. The image bakes a non-root slam user, which rootless docker maps to a foreign host subuid that cannot write host-owned rw mounts — so the nodeset declares server_container_user = "0:0": under rootless, container-root is the invoking host user, so the handle drop stays a normal 755 directory and every .npz lands on the host owned by the user (HOME=/home/slam rides server_env because the image's venv/cache paths assume the baked user's home). The registry also creates a missing outputs/pyslam_maps/ automatically — no manual setup.

Weights are external. Neither image bakes in model weights — they carry only code and compiled kernels. The multi-view checkpoints live in a host data/models/pyslam/ folder and mount read-only into the container via server_mounts (PYSLAM_WEIGHTS_DIR overrides the location; missing dirs are skipped, that backend just errors at call time); the HuggingFace-runtime backends download into a mounted rw cache on first use. Fetch the explicit checkpoints with the tracked scripts under docker/weights/:

cd workspace/nodesets/model/model_pyslam/docker/weights
bash download_mast3r.sh      # MASt3R (~2.5GB) — the verified default
bash download_all.sh         # mast3r + mvdust3r explicit checkpoints

export DOCKER_HOST="unix:///run/user/$(id -u)/docker.sock"
bash workspace/nodesets/model/model_pyslam/docker/build.sh agentcanvas/pyslam:cpu-fixed

Run the TUM showcase graph through the experiment wrapper (profile pyslam-tum, CPU-only):

# One episode of the TUM showcase, submitted to the slot backend.
# Container Launch fans out one container per eval worker, so worker_count
# is no longer pinned to 1 by the nodeset (the map is per-worker state).
AGENTCANVAS_BACKEND_URL=http://127.0.0.1:8001 \
  python .claude/commands/experiment/bin/submit.py \
  pyslam-tum pyslam_tum_slam \
  split=freiburg1 episode_count=1 worker_count=1 step_budget=60

7. Environment Variables

All PYSLAM_* variables are snapshotted from the host environment when the backend imports the nodeset (the class-level server_env) and injected into the container via -e — set them before starting the backend. Per-node config fields on the Reset node (sensor type, presets, HFOV, depth scale) override these env defaults.

8. Operational notes

• Bootstrap latency. RGB-D SLAM needs a few frames of parallax to initialise. On a validated Habitat run the front-end spent ~16 discrete steps in NOT_INITIALIZED before locking on, then tracked steadily — reconstructing a metric room-scale cloud (~1.5–1.9k points, core extent ≈ 5 × 3 × 7 m) matching the walked path.
• Coverage is episode-dependent. Episodes dominated by in-place rotation, very short paths, or low-texture views may never accumulate enough parallax to initialise, and legitimately return zero map points — an inherent property of feature SLAM under discrete large-rotation motion, not a fault. The front-end honestly reports "not initialised" rather than emitting spurious geometry.
• Per-episode reset is clean. Across a multi-episode run each episode re-bootstraps from an empty map (reset_session); maps never carry over, and a failed-to-initialise episode exports an empty cloud rather than the previous episode's points.