HM-EQA NodeSet

The HM-EQA NodeSet (EnvHMEQANodeSet) wraps Habitat-Sim + HM3D semantic meshes for House-Mesh Embodied Question Answering (Ren et al. 2024, explore-eqa). Each episode is a question about a specific scene+floor; the agent explores via free-pose teleports and emits a letter answer (A/B/C/D).

Sibling of Habitat — not an extension. The two cannot share an environment: Habitat NodeSet pins habitat-sim 0.1.7 + Python 3.8 (VLN-CE); HM-EQA uses a newer habitat-sim + Python 3.9 + torch 2.2.1. Different conda env (hmeqa), different simulator instance.

1. Overview

Purpose

House-Mesh Embodied Question Answering asks multi-choice questions over HM3D semantic scenes. Each episode = one question + one scene + one floor + one initial pose. The 500-question val split ships in the HM-EQA repo's CSVs (HM3D train-split license gates the training Qs).

Architecture

Three-layer pattern mirroring Habitat:

1. HMEQAEnvManager (singleton) — per-episode simulator rebuild (one .glb per episode), pinned single-thread executor for GL affinity, loads questions.csv + scene_init_poses.csv once.
2. Canvas tool nodes — 5 tools (reset / step / episode_info / cam_intrinsics / evaluate).
3. EnvHMEQANodeSet — server_python points to $HMEQA_PYTHON, so the framework auto-routes the nodeset into a ac-hmeqa subprocess when loaded via ?mode=server.

Server-Mode Only

The nodeset always runs server-mode: habitat-sim isn't installed in the default agentcanvas env. Set HMEQA_PYTHON=/home/$(whoami)/miniforge3/envs/ac-hmeqa/bin/python before loading, or rely on the default.

2. Canvas Nodes

Output-port highlights

• cam_pose_matrix (ANY): 4×4 extrinsic in the TSDF frame (already composed via pose_habitat_to_normal → pose_normal_to_tsdf). Feeds directly into TSDFPlanner.integrate.
• pose_normal (ANY): 3-vector agent position in normal frame. Feeds TSDFPlanner.find_next_pose.
• tsdf_bnds (ANY): [[x_lo, x_hi], [y_lo, y_hi], [z_lo, z_hi]] derived from the HM3D mesh bounds + floor height; constructs the planner on first step.
• num_step (ANY): scene-size-driven budget — int(sqrt(scene_size) * max_step_room_size_ratio), typically 10–20 steps.

3. Action Contract

Free-pose teleport, sent as JSON TEXT:

{"position_normal": [x, y], "angle": 1.5708}

• position_normal: [x, y] in the normal frame. The env appends the episode's floor_height and converts to habitat coordinates internally.
• angle: yaw in radians.

There is no mid-episode "answer" action. Answer emission is post-hoc on the method side — see ExploreEQA NodeSet aggregate step. This deliberately sidesteps the action-manifest unification tracked as TODO #46 — once a second multi-action env shows up we'll bundle them.

Source reference: workspace/nodesets/_upstream/explore-eqa/upstream/run_vlm_exp.py:313-321.

4. Observation Bundle

Every reset and step returns the same 10/11-port bundle. The bundle is designed so a downstream method nodeset (e.g. ExploreEQA) can run TSDFPlanner.integrate, find_prompt_points_within_view, and find_next_pose without needing any env-side knowledge.

Black-image guard

The env tracks is_black in the bundle when the agent falls through the floor / outside the mesh. The HM-EQA method nodeset skips TSDF integration on black frames (matching upstream behaviour).

5. Configuration

Controlled via module-level _DEFAULTS in workspace/nodesets/env/env_hmeqa/__init__.py. Override via env vars if needed:

Paper-baseline values are mirrored from workspace/nodesets/_upstream/explore-eqa/upstream/cfg/vlm_exp.yaml.

Data layout (ADR-platform-005):

data/hm3d/hmeqa/questions.csv           HM-EQA Q&A
data/hm3d/hmeqa/scene_init_poses.csv    per-(scene, floor) init
data/hm3d/hmeqa/Open_Sans/              annotation font (method-side)
data/hm3d/hm3dsem/{scene}/{scene[6:]}.basis.glb       meshes (HuggingFace-gated)
data/hm3d/hm3dsem/{scene}/{scene[6:]}.basis.navmesh   navmesh

Environment variable overrides:

• HMEQA_PYTHON — hmeqa-env Python interpreter path (default /home/$(whoami)/miniforge3/envs/ac-hmeqa/bin/python)
• HMEQA_DATA_ROOT — override the CSV/font dir
• HMEQA_SCENE_ROOT — override the HM3D mesh dir

6. Env panel Integration

HMEQAEnvPanel is a two-field cascade:

Actions: play, pause, stop, reset. The play button triggers run_start; Reset re-applies the currently-selected episode without touching graph state.

The split field is retained as a single-option select to give future train/test splits a zero-migration path.

7. Environment Setup

One-time install:

bash scripts/install/install_ac_hmeqa.sh      # creates the hmeqa conda env
bash scripts/data/fetch_episodes_vln.sh --hmeqa # stages CSVs + fonts

HM3D mesh download is manual — gated by HuggingFace auth:

1. Request access at https://aihabitat.org/datasets/hm3d-semantics
2. huggingface-cli login with a read token
3. Download hm3d-train-habitat-v0.2.tar and extract to data/hm3d/hm3dsem/

See also:

• ExploreEQA NodeSet — the method side that consumes this env
• Habitat NodeSet — the VLN-CE sibling