HM-EQA NodeSet

The HM-EQA NodeSet (EnvHMEQANodeSet) wraps Habitat-Sim + HM3D semantic meshes for House-Mesh Embodied Question Answering (Ren et al. 2024, explore-eqa). Each episode is a question about a specific scene+floor; the agent explores via free-pose teleports and emits a letter answer (A/B/C/D).

Sibling of Habitat — not an extension. The two cannot share an environment: Habitat NodeSet pins habitat-sim 0.1.7 + Python 3.8 (VLN-CE); HM-EQA uses a newer habitat-sim + Python 3.9 + torch 2.2.1. Different conda env (hmeqa), different simulator instance.

1. Overview

Purpose

House-Mesh Embodied Question Answering asks multi-choice questions over HM3D semantic scenes. Each episode = one question + one scene + one floor + one initial pose. The 500-question val split ships in the HM-EQA repo's CSVs (HM3D train-split license gates the training Qs).

Architecture

Three-layer pattern mirroring Habitat:

1. HMEQAEnvManager (singleton) — per-episode simulator rebuild (one .glb per episode), pinned single-thread executor for GL affinity, loads questions.csv + scene_init_poses.csv once.
2. Canvas tool nodes — 5 tools (reset / step / episode_info / cam_intrinsics / evaluate).
3. EnvHMEQANodeSet — server_python points to $HMEQA_PYTHON, so the framework auto-routes the nodeset into a ac-hmeqa subprocess when loaded via ?mode=server.

Server-Mode Only

The nodeset always runs server-mode: habitat-sim isn't installed in the default agentcanvas env. Set HMEQA_PYTHON=/home/$(whoami)/miniforge3/envs/ac-hmeqa/bin/python before loading, or rely on the default.

2. Canvas Nodes

Output-port highlights

• cam_pose_matrix (ANY): 4×4 extrinsic in the TSDF frame (already composed via pose_habitat_to_normal → pose_normal_to_tsdf). Feeds directly into TSDFPlanner.integrate.
• pose_normal (ANY): 3-vector agent position in normal frame. Feeds TSDFPlanner.find_next_pose.
• tsdf_bnds (ANY): [[x_lo, x_hi], [y_lo, y_hi], [z_lo, z_hi]] derived from the HM3D mesh bounds + floor height; constructs the planner on first step.
• num_step (ANY): scene-size-driven budget — int(sqrt(scene_size) * max_step_room_size_ratio), typically 10–20 steps.

3. Action Contract

Free-pose teleport, sent as JSON TEXT:

{"position_normal": [x, y], "angle": 1.5708}

• position_normal: [x, y] in the normal frame. The env appends the episode's floor_height and converts to habitat coordinates internally.
• angle: yaw in radians.

There is no mid-episode "answer" action. Answer emission is post-hoc on the method side — see ExploreEQA NodeSet aggregate step. This deliberately sidesteps the action-manifest unification tracked as TODO #46 — once a second multi-action env shows up we'll bundle them.

Source reference: workspace/nodesets/_upstream/explore-eqa/upstream/run_vlm_exp.py:313-321.

4. Observation Bundle

Every reset and step returns the same 10/11-port bundle. The bundle is designed so a downstream method nodeset (e.g. ExploreEQA) can run TSDFPlanner.integrate, find_prompt_points_within_view, and find_next_pose without needing any env-side knowledge.

Black-image guard

The env tracks is_black in the bundle when the agent falls through the floor / outside the mesh. The HM-EQA method nodeset skips TSDF integration on black frames (matching upstream behaviour).

5. Configuration

Controlled via module-level _DEFAULTS in workspace/nodesets/env/env_hmeqa/__init__.py. Override via env vars if needed:

Paper-baseline values are mirrored from workspace/nodesets/_upstream/explore-eqa/upstream/cfg/vlm_exp.yaml.

Data layout (ADR-platform-005):

data/hm3d/hmeqa/questions.csv           HM-EQA Q&A
data/hm3d/hmeqa/scene_init_poses.csv    per-(scene, floor) init
data/hm3d/hmeqa/Open_Sans/              annotation font (method-side)
data/hm3d/hm3dsem/{scene}/{scene[6:]}.basis.glb       meshes (HuggingFace-gated)
data/hm3d/hm3dsem/{scene}/{scene[6:]}.basis.navmesh   navmesh

Environment variable overrides:

• HMEQA_PYTHON — hmeqa-env Python interpreter path (default /home/$(whoami)/miniforge3/envs/ac-hmeqa/bin/python)
• HMEQA_DATA_ROOT — override the CSV/font dir
• HMEQA_SCENE_ROOT — override the HM3D mesh dir

6. Env panel Integration

HMEQAEnvPanel is a two-field cascade:

Actions: play, pause, stop, reset. The play button triggers run_start; Reset re-applies the currently-selected episode without touching graph state.

Superseded 2026-07-29: the cascade is now three-deep — dataset (hmeqa / mt_hm3d) → split → episode_index — since the dataset-spec extension in §8. Eval submissions select MT-HM3D with dataset=mt_hm3d, like ObjectNav's mp3d_v1.

7. Environment Setup

One-time install:

bash scripts/install/install_ac_hmeqa.sh      # creates the hmeqa conda env
bash scripts/data/fetch_episodes_vln.sh --hmeqa # stages CSVs + fonts

HM3D mesh download is manual — gated by HuggingFace auth:

1. Request access at https://aihabitat.org/datasets/hm3d-semantics
2. huggingface-cli login with a read token
3. Download hm3d-train-habitat-v0.2.tar and extract to data/hm3d/hm3dsem/

See also:

• ExploreEQA NodeSet — the method side that consumes this env
• Habitat NodeSet — the VLN-CE sibling

8. Datasets: hmeqa / mt_hm3d

Since 2026-07-29 the nodeset carries a _DATASET_SPECS table and serves two benchmarks through one env. Selecting a dataset is an episode-corpus choice, not a task change — the verified explore_eqa_hmeqa graph runs MT-HM3D unmodified via the panel's dataset field (or dataset=mt_hm3d on an eval submission).

Each spec names its own questions/init-pose CSVs, its own choices parser, and a force_floor0_init flag. Three upstream behaviours are deliberate benchmark-native mirrors — do not "fix":

• MT's init-pose key forces floor 0 (memory_eqa.py:165) even though 260/1,587 questions target floor 1/2.
• MT choices parse with MemoryEQA's strip/split(", ") (memory_eqa.py:167) — differs from explore-eqa's parser on 38 rows and from literal_eval too.
• The evaluated MT-HM3D-filtered*.csv from the paper is not public; we run the 1,587-row contextual release and state that口径 when reporting.

MT-HM3D verification evidence

Not yet: any multi-episode MT-HM3D number; the MemoryEQA method itself (YOLOv11 + CLIP + RAG memory — reference clone at tmp/thirdparty/MemoryEQA). The sibling benchmark from the same lineage, EXPRESS-Bench, is a separate nodeset — see EXPRESS-Bench.