ORB-SLAM3 NodeSet
Classic feature-based RGBD visual SLAM — the first Container-Launch nodeset
model_orbslam3 wraps ORB-SLAM3 — the de-facto baseline of feature-based visual SLAM — as five canvas nodes: reset / track / get_trajectory / get_status / save_trajectory. It is the first nodeset launched via Container Launch: server_image = "agentcanvas/orbslam3:latest" is the entire deployment declaration — no conda env, no bridge code, no per-nodeset client/server. Verified end-to-end on TUM fr1_xyz (2026-08-13): 60/60 frames tracked OK through the full framework path.
orbslam3_tum_slam graph): env_tum streams a TUM sequence, the containerized ORB-SLAM3 session tracks each frame, and the observation + trajectory viewers update per step until the run completes.1. Deployment: one ClassVar, zero bridge code
The compiled ORB-SLAM3 stack (C++ core, boost-python binding, ORB vocabulary) lives in the docker image agentcanvas/orbslam3:latest (3.81 GB; build recipe in workspace/nodesets/model/model_orbslam3/docker/, based on the Hello Robot fork pair hello-binit/ORB_SLAM3 + ORB_SLAM3-PythonBindings). The nodeset itself is ordinary Python in the repo — the container runs the stock auto_host from the read-only repo mount, so this same file declares the nodes to the backend at scan time and executes them inside the container at run time. Contrast with pySLAM, whose pre-framework container bridge needed a hand-written _client.py/_server.py pair; that mechanism is now framework-owned.
| Declaration | Value | Effect |
|---|---|---|
server_image | agentcanvas/orbslam3:latest | Container Launch trigger; auto-routes to server mode |
server_image_gpu | False | ORB-SLAM3 core is CPU-only |
server_mounts | {"outputs/slam_runs": "/opt/out"} | Writable drop for trajectory-file handles (host-visible; verified under rootless docker) |
parallelism | "replicated" | Stateful SLAM session — one container per eval worker |
expected_ram_mb | 1500 | Admission preset (vocabulary + atlas, steady state) |
2. A streaming session, not a stateless pass
SLAM accumulates state: the map, keyframes, and loop-closure database live in one orbslam3.System instance held as a module-level singleton inside the container's long-lived auto_host process. reset (re)creates the session from camera intrinsics (rendering the ORB-SLAM3 settings YAML on the fly); every track fire feeds one RGBD frame into the same session; the query nodes read it. Fire reset once per episode — multi-worker eval gives each worker its own container, so sessions never collide.
3. Canvas nodes
| Node | In → Out | Backing API |
|---|---|---|
model_orbslam3__reset | intrinsics → ready | System(voc, settings, Sensor.RGBD); config depth_scale converts input depth to metres (VLN-CE normalised depth → 10.0) |
model_orbslam3__track | rgb, depth, timestamp → pose, tracking_ok, num_frames_ok | process_image_rgbd + get_tracking_state; pose is None while not OK |
model_orbslam3__get_trajectory | trigger → trajectory, num_poses | get_trajectory_points (loop-closure-corrected; 13-tuple entries → 4×4 matrices) |
model_orbslam3__get_status | trigger → status | tracking state + frame counters; safe before reset |
model_orbslam3__save_trajectory | trigger → handle | TUM-format text into the writable mount; returns the host-side path |
4. Verification record (2026-08-13)
| Check | Result |
|---|---|
| Binding surface (in-container probe) | System/Sensor/TrackingState confirmed; trajectory entry = 13-tuple (ts + row-major 3×4); Camera.fps must be an integer; Viewer settings block is required even headless |
| TUM fr1_xyz, 60 frames, in-container | 60/60 tracked OK (RGBD initializes on frame 1); map 1049 points |
Full framework path (ContainerServer → auto_host → msgpack /call) | manifest 5 nodes; reset → track×60 (60/60 OK) → get_trajectory (60 poses) → save_trajectory handle readable on host; clean stop, no leftover container |
| Trajectory sanity | 0.39 m max displacement over 2.0 s — consistent with fr1_xyz handheld motion |
Operational note: System.shutdown() can hang joining ORB-SLAM3's background threads after longer sessions (observed after a 300-frame run; the 60-frame smoke shut down cleanly). Under Container Launch this is contained — ContainerServer.stop() goes through docker stop -t 10, which SIGKILLs the container after the grace window — but a hung shutdown inside a long-running graph eval will eat those 10 seconds per unload.
Also verified (2026-08-13, later the same day): registry-level load (POST /api/components/nodesets/model_orbslam3/load → mode=server, container spawned, 5 proxy nodes registered) and a full canvas graph run — workspace/graphs/vln/unverified/orbslam3_tum_slam.json (env_tum replay → reset/track per step → viewers → get/save trajectory; 296-pose TUM-format file landed in the writable mount). A Habitat companion probe (orbslam3_slam_probe.json, CMA policy + SLAM observer, depth_scale=10.0) is authored but not yet run. Multi-worker fan-out remains unexercised.
5. Usage
Build the image once: bash workspace/nodesets/model/model_orbslam3/docker/build.sh (≈14 min). Load: POST /api/components/nodesets/model_orbslam3/load — a missing image fails fast with an actionable error. Feed track from any RGBD env; set reset.depth_scale = 10.0 for VLN-CE-style normalised depth, 1.0 for metric depth.