Matterport3D (MP3D) NodeSet

The Matterport3D NodeSet (EnvMP3DNodeSet, node prefix env_mp3d) wraps MatterSim — the discrete panoramic navigation simulator behind R2R and its sibling VLN datasets — as a set of pure environment tools. This page is about how the environment is abstracted: the decoupled env/method split, the three-tier env contract that lets agent graphs port across simulators, and the per-view image primitive that replaced the old composite panorama.

1. The abstraction in one picture

MatterSim is a stateful C++ simulator: it holds one scene mesh + one agent pose, renders a camera frame per makeAction, and exposes a connectivity graph of viewpoints per scene. AgentCanvas hides all of that behind a NodeSet whose job is deliberately narrow:

• The env emits primitives, the method does the reasoning. env_mp3d renders pixels, lists navigable neighbours, teleports between viewpoints, and scores a trajectory. It does no prompt formatting, no LLM calls, no compass prose, no vision captioning. Those live in method nodesets (NavGPT-MP3D, MapGPT, DiscussNav, SpatialNav) that consume the env's primitives. This is the project's decoupled_nodesets rule — it is what lets the same MapGPT graph swap MP3D for a future env by renaming node_type prefixes.
• Graph mode, not continuous control. R2R navigation is discrete: the agent picks one of the current viewpoint's navigable neighbours and "teleports" there along the connectivity graph. env_mp3d exposes exactly that — an action is a target viewpoint ID (or "STOP"), never a velocity/turn. A continuous-control tier once existed and was deleted as dead weight (see Changelog).
• One nodeset, six dataset schemas. R2R · R4R · RxR · REVERIE · CVDN / NDH all ride the same MatterSim connectivity graph; they differ only in episode metadata. The nodeset surfaces them through a single dataset → split → episode_index cascade on MP3DEnvPanel.

MP3DEnvManager singleton

Like every server-mode env, all simulator state lives in a process-wide singleton with a dedicated single-threaded executor (GL/sim thread affinity). Canvas nodes never touch MatterSim directly — they marshal blocking calls onto the executor:

mgr = MP3DEnvManager.get()           # always the same instance
pano = await loop.run_in_executor(mgr.executor, mgr.graph_render_panorama, n_headings)
obs  = await loop.run_in_executor(mgr.executor, mgr.graph_get_observation)

Because the simulator is stateful (per-worker scene + agent pose), the nodeset declares parallelism = "replicated": each eval worker gets its own subprocess + sim instance rather than sharing one.

2. The three-tier env contract

Every env NodeSet (Habitat, MP3D, future AI2-THOR) follows the same tiered shape so that agent graphs wired only to the required ports are env-portable. env_mp3d populates all three tiers:

The gym facade is sugar: reset = episode_info + graph_panorama + observation, and step = navigate_to + graph_panorama + observation, fused into one node so the downstream agent sees an identical observation shape at step 0 and every step after.

3. The per-view image primitive

This is the load-bearing piece of the abstraction and the one most recently reshaped. MatterSim renders a panorama as N headings × 3 elevations discrete views (default 12 × 3 = 36, reproducing the NavGPT sweep with elevations −30°/0°/+30°). The env now emits those views as a list, never as a stitched grid image:

The contract-required observation port is just views under its portable name. A built-in imageViewer renders LIST[IMAGE]/LIST[DEPTH] as a tiled strip, so the panorama is still inspectable on the canvas without the env building a composite.

Why per-view instead of a composite panorama

The env used to stitch the 36 views into a single horizontal grid image (composite / depth_composite) plus a directions string. That pushed image-handling logic back into every method: MapGPT, the four NavGPT perception nodes, and DiscussNav each hard-coded the grid geometry (border, label height, tile layout) to crop the panorama apart again. The round-trip was lossy and contract-violating:

• Elevation was thrown away — the composite was a single horizontal strip, so methods could only recover the 0° row and silently lost the up/down views.
• Candidate→image was a heading-nearest crop, not an exact index, so a navigable neighbour could be matched to the wrong tile.
• Stitch-then-crop introduced pixel drift and made every method brittle to the env's layout constants.

Inverting the contract — env hands out primitives, methods index into them — deleted all three crop paths and let each candidate carry an exact view_index (see next section). This is the same env-emits-primitives principle from §1, applied to pixels.

Candidate → view_index mapping

On reset/step, every entry in navigable_json gains a view_index field: the rendered view whose (heading, elevation) is nearest that candidate's direction. With 36 views the match lands on the correct elevation row, so a method can pull the exact image for a candidate by index — no cropping, no heading guesswork. Graph-granularity methods that don't need per-candidate imagery (NavGPT-MP3D, SpatialNav) simply ignore the field.

4. Canvas nodes

get_tools() returns seven nodes — two gym-facade, five graph-tier primitives:

Render granularity is a node config (n_headings: 8 → 24 views, or 12 → 36 views, default 12) on reset, step, and graph_panorama.

Navigation & trajectory accounting

navigate_to is pure graph movement. An adjacent target is a single step; a non-adjacent target walks the shortest path and records every intermediate viewpoint. The full trajectory is written to graph_state under trajectory — that is what evaluate reads to compute SR / SPL / nDTW / SDTW / CLS / nav_error / oracle_success against the ground-truth path. One quirk worth knowing: the done output key is emitted only on STOP, to keep the executor's done-scan from terminating the episode mid-walk.

5. Env panel & datasets

MP3DEnvPanel is the canvas panel for episode selection. It exposes a three-field cascade — dataset → split → episode_index — plus Play / Pause / Stop / Reset actions. Changing dataset resets split + episode; changing split resets episode; any change fires an episode_reset signal so lifetime="episode" state containers downstream clear themselves.

All six are surfaced through the one nodeset — the env panel cascade picks the schema, and dataset-specific fields ride along in extras_json so the contract ports stay uniform.

6. Server mode

MatterSim has its own native dependency stack, so the nodeset runs as an auto-hosted subprocess under a dedicated interpreter (MP3D_PYTHON, default ~/miniforge3/envs/ac-mp3d/bin/python). When loaded via POST /api/components/nodesets/env_mp3d/load?mode=server, the framework spins up the subprocess and proxies the nodes, preserving their node_type IDs (e.g. env_mp3d__step) so graphs run identically regardless of deployment mode.

Episode state lives inside the subprocess (the main-backend singleton is empty under the active-workspace overlay). Episode browsing from the canvas therefore goes through the env panel / /api/env-panels/env_mp3d/..., not through hidden tool nodes. Initialization kwargs (passed to initialize() at subprocess boot): width, height, vfov, preloading, depth, dataset, split, cache_size.

7. Typical graph workflow

A discrete LLM-VLN agent on MP3D wires the gym facade into a loop:

env_mp3d__reset ──┬─ instruction ───────────────┐
                  ├─ views / view_meta ──► (method: format candidates + images)
                  └─ navigable_json ────────────┘
                                                  │
                          (method LLM picks a viewpoint ID)
                                                  ▼
                              env_mp3d__step(action) ──┬─ views / view_meta / navigable_json ─► loop
                                                       └─ done ─► env_mp3d__evaluate ─► metrics

• MapGPT consumes views + view_meta and pulls each candidate's image by its view_index.
• NavGPT-MP3D feeds views to BLIP-2 / Faster R-CNN perception nodes, which bucket detections into compass sectors by the heading_deg/elevation_deg in view_meta (no grid geometry assumed).
• SpatialNav adds SSG enrichment on top of the same view stream.

For the live status of each MP3D graph (verified vs. needs-investigation), see VLN Support Status §2.2. For the env-contract rationale across all environments, see NodeSets design doc.