Habitat NodeSet

The Habitat NodeSet (EnvHabitatNodeSet) exposes the Habitat-Sim VLN-CE environment as a unified suite of canvas tools for navigation and observation. It manages the simulator lifecycle, handles both local and server-mode execution, and provides tools for stepping through episodes, observing the environment, and controlling episode state.

1. Overview

Purpose

The Habitat NodeSet wraps the Habitat-Sim 0.1.7 VLN-CE (Vision-and-Language Navigation, Continuous Embodied) simulator. It enables graphs to: - Step through navigation episodes with RGB + depth observation - Query agent position and orientation - Control episode lifecycle (switch episodes, change splits) - Render panoramic multi-view composites for scene understanding

Architecture

The nodeset is organized around a singleton HabitatEnvManager that: - Maintains the single Habitat environment instance - Enforces thread safety via locks - Uses a dedicated single-threaded executor for GL/physics affinity - Exposes blocking methods safe for async/sync code boundaries

Local vs Server Mode

When loaded via POST /api/components/nodesets/env_habitat/load?mode=server, the framework spins up a server subprocess using the path specified in server_python. The proxy nodes preserve the original node_type IDs (e.g., env_habitat__step), so graphs work identically regardless of deployment mode.

HabitatEnvManager Singleton

mgr = HabitatEnvManager.get()  # Always returns the same instance
mgr.initialize(exp_config, split, gpu_id, max_steps)
obs = mgr.step(action)  # Returns dict with rgb, depth, state, done

All public methods are blocking and thread-safe. Async code calls them via:

await asyncio.get_running_loop().run_in_executor(mgr.executor, mgr.step, action)

2. Canvas Nodes

User-Facing Tools

These nodes appear in the canvas sidebar and are safe to use in graphs:

This nodeset implements the gym-like env interface (4 verbs + pull perception, 2026-06-09). step_* nodes return no observation — perception is pulled via the observe_* family. The old step_native / observe / localize / episode_info / panorama_rgbd nodes and the two [Mock] nodes were retired; see the template's migration map.

Notable Output Ports

• pose (POSE wire type): {"position": [x, y, z], "orientation": [qx, qy, qz, qw]} — emitted by observe_egocentric.
• raw_obs (ANY): Raw Habitat observation dict; used by policy nodes that need full sensor data.
• terminated / truncated (BOOL): gym-style episode-end split — STOP/goal vs step-budget cutoff.
• info (ANY): per-step diagnostics + terminal metrics dict (populated when terminated/truncated).
• composite (IMAGE) / views (ANY): the two panorama representations from observe_panorama.

Internal Tools (Episode Control, ADR-legacy-002)

These tools are hidden from the sidebar (category="internal") but auto-proxied when the nodeset runs in server mode. Canvas evaluation env panels use them to manage episodes:

3. Configuration

Initialization Parameters

When loading the nodeset via POST /api/components/nodesets/env_habitat/load, pass these kwargs in the request body:

{
  "dataset": "R2R-CE",
  "split": "val_unseen",
  "gpu_id": 0,
  "max_steps": 500
}

Environment Variables

# Path to vlnce conda environment (Python 3.8)
export VLNCE_PYTHON="python"

# VLN-CE root directory (if not found via discovery)
export VLNCE_ROOT="/path/to/VLN-CE"

The nodeset auto-discovers VLN-CE in this order: 1. $VLNCE_ROOT/data if set 2. ../VLN-CE/data (relative to agentcanvas repo root) 3. third_party/VLN-CE/data

If found, the process cwd is changed to the VLN-CE root so Hydra configs resolve correctly.

Node-Level Config: Panorama View Count

The Panorama tool exposes a UI config field:

ConfigField("n_views", "select", label="Views", default=12,
            options=[
                {"value": 4, "label": "4 views (90°)"},
                {"value": 8, "label": "8 views (45°)"},
                {"value": 12, "label": "12 views (30°)"},
                {"value": 24, "label": "24 views (15°)"},
            ])

Users can adjust view density in the node inspector before firing.

4. Datasets: R2R-CE and RxR-CE

The Habitat NodeSet supports two VLN-CE benchmarks under a single unified env panel:

4.1 Language-Suffixed Splits (RxR-CE)

RxR-CE episodes carry a BCP-47 language field per instruction (e.g. en-US, hi-IN, te-IN). The env panel exposes each base split three times, one per language group, following the same convention as the MP3D discrete nodeset:

val_unseen_en   (matches en-US, en-IN)
val_unseen_hi   (matches hi-IN)
val_unseen_te   (matches te-IN)

Behind the scenes the suffix selects the matching rxr_cma_{en,hi,te}.yaml upstream config — each YAML already sets EVAL.LANGUAGES to the right BCP-47 list, which HabitatEnvManager.initialize threads into TASK_CONFIG.DATASET.LANGUAGES so RxRVLNCEDatasetV1's loader filters correctly at load time.

4.2 Episode Extras

Every episode info dict (from _get_episode_info_unlocked, get_episodes_list, and _peek_episode_sync) now carries:

The same extras schema is emitted by the MP3D nodeset (matterport3d.py::_load_rxr), so downstream agent graphs can consume extras.language / extras.instruction_id uniformly across simulators.

4.3 Data Download

Episodes for R2R-CE and RxR-CE land under data/habitat/datasets/ via scripts/data/fetch_episodes_vln.sh:

# RxR-CE episodes (~150 MB, via gdown from the upstream Google Drive)
bash scripts/data/fetch_episodes_vln.sh --rxr-ce

The flag is idempotent — if every expected {split}/{split}_guide.json.gz file is already on disk, it no-ops. Without gdown, it prints a clear manual-download hint pointing at the official RxR-Habitat Challenge zip.

Out of scope (tracked separately): pre-computed BERT instruction features (E7) needed only for the CMA policy baseline, and pose-trace trajectories.json.gz preprocessing (E8) needed only for VLN-CE's recollect trainer.

5. Env Panel Integration

ADR-server-002: Episode Lifecycle Management

The Habitat NodeSet declares a HabitatEnvPanel (a BaseEnvPanel subclass) as a class-level attribute. This enables the canvas Evaluate page to manage episodes uniformly across all environments:

# On EnvHabitatNodeSet (registered by WorkspaceComponentRegistry on load)
env_panel = HabitatEnvPanel

The env panel provides: - Metadata queries (splits, episode counts) via on_load() / get_options() - Episode switching by index via on_field_change("episode_index", …) - Split switching with full teardown + rebuild - Run lifecycle buttons (play / pause / stop / reset) via on_action()

How It Works

1. Evaluate page calls GET /api/env-panels/env_habitat/state → splits + current state from on_load()
2. User selects episode from dropdown → POST /api/env-panels/env_habitat/field/episode_index
3. HabitatEnvPanel.on_field_change calls mgr.set_episode(…) and emits the episode_reset signal
4. Graph continues; agent navigates in the new episode

This abstraction allows the same Evaluate UI to work with Habitat, SAPIEN, or any future environment that ships a BaseEnvPanel subclass. In server mode the panel is hosted inside the env subprocess and bridged over /env-panel/* via RemoteEnvPanelProxy.

Three-Field Cascade (Dataset → Split → Episode)

The HabitatEnvPanel exposes a three-field cascade mirroring MP3DEnvPanel:

• dataset — R2R-CE or RxR-CE. Changing resets split and episode_index.
• split — dataset-conditional list via _dataset_splits(dataset). Changing triggers _switch_split, which resolves the (exp_config, base_split) pair via _resolve_dataset_config, shuts the simulator down, and re-initializes against the new YAML.
• episode_index — index into the currently loaded dataset/split.

Each field change emits a signal side-effect with signal_name="episode_reset", so any lifetime="episode" state container clears downstream. This matches the MP3D nodeset contract so agent graphs can bind state lifetimes without knowing which simulator is active.

6. Usage

Loading the NodeSet

Local mode:

POST /api/components/nodesets/env_habitat/load

Server mode (explicit):

POST /api/components/nodesets/env_habitat/load?mode=server

Auto-routed (default):

POST /api/components/nodesets/env_habitat/load

Typical Graph Workflow

1. Load Habitat nodeset
   ↓
2. Habitat: Episode Info → get instruction
   ↓
3. Loop:
   - Habitat: Observe → current RGB + depth
   - [Policy or reasoning node] → decide action
   - Habitat: Step → action=[0-3], get new obs + done
   - [Iterate or break if done]
   ↓
4. Final metrics available when done=True

Common Patterns

Panoramic Scene Understanding

Habitat: Observe → current RGB
   ↓ (optionally wire agent_state)
Habitat: Panorama (n_views=12) → composite grid + directions JSON
   ↓
Vision model for multi-view analysis

Navigation with Shortest Path

Habitat: Episode Info → get current position
   ↓
[Planner determines target] → target_state={position: [x,y,z]}
   ↓
Habitat: Navigate To → done, final_state

Episode Switching in Evaluation

Use the Evaluate page's Episode dropdown. Behind the scenes: 1. Evaluator calls env_habitat__ep_metadata → get split + count 2. User selects episode index 3. Evaluator calls env_habitat__ep_set with index 4. Graph resets and continues in new episode

Error Handling

• "Environment not initialized": Habitat nodeset not loaded; load it first
• "Episode already done": Auto-reset on next Observe or explicitly call Episode Info → Set Episode
• "Index out of range": Requested episode index exceeds dataset size; check Episode List
• Split switch fails: Ensure VLN-CE is at the expected path or set $VLNCE_ROOT

Performance Considerations

• Single simulator: All graph executions share one Habitat instance (thread-safe via locks)
• Thread affinity: GPU/physics operations run on dedicated executor thread
• Observation caching: Observe returns raw_obs dict; extract specific keys (rgb, depth) as needed
• Panorama rendering: N_views=24 takes ~2-3s per panorama on typical GPU; use 4 or 8 for quick iteration