OpenEQA NodeSet (EM-EQA)

The OpenEQA NodeSet (EnvOpenEQAEMNodeSet) wraps the Episodic-Memory mode of OpenEQA (Majumdar et al., CVPR 2024) — a free-form Embodied Question Answering benchmark scored by an LLM-as-judge on a 1–5 scale (LLM-Match).

Sibling of HM-EQA, not an extension. HM-EQA is multi-choice with token-likelihood scoring (Prismatic VLM); OpenEQA is free-form text with an external LLM judge. They share the ac-hmeqa conda env in server mode but are independent nodesets.

v1 scope: EM-EQA only. The active-exploration mode (A-EQA) is tracked as roadmap E10 — it would reuse the HM-EQA Habitat manager + an explore-eqa-style termination policy on top of these tools.

1. Overview

Purpose

EM-EQA gives the agent a fixed set of pre-recorded frames from a walkthrough of a 3D scene plus a free-form question (e.g. "What is on top of the kitchen counter?"). The agent answers in natural language; an LLM judge scores 1–5 by comparing against a reference answer. The benchmark's headline metric, LLM-Match, normalizes that score to [0, 1] via (score − 1) / 4.

There is no interactive simulator in EM-EQA — observation is a fixed list of images, not a step()-able sim. The OpenEQA nodeset is therefore a pure asset+judge surface; it does not export step or done.

Architecture

Three-layer pattern mirroring the other env nodesets:

1. OpenEQAEMManager (subprocess-singleton) — loads open-eqa-v0.json once at initialize(), then on reset() returns the episode's frame file paths as a plain list[str] (no decode). PNG decoding is deferred to sample_frames, which decodes only the K frames actually consumed by the VLM — for ScanNet (600 frames @ 1296×968) at K=50 this drops per-episode peak RSS from ~2.2 GB to ~190 MB and reset wall-time from ~19 s to under 10 ms. The wire payload between reset → sample_frames is a JSON-serialisable string list (port type ANY), which is what makes the lazy contract survive the server-mode RPC.
2. Canvas tool nodes — 5 tools (reset / episode_info / sample_frames / parse_score / emit_metrics). The judge LLM call itself is the framework's builtin llmCall; the OpenEQA-specific score parsing + metric naming live in the two thin nodeset nodes.
3. EnvOpenEQAEMNodeSet — server_python defaults to $HMEQA_PYTHON, so the framework auto-routes the nodeset into a ac-hmeqa subprocess when loaded via ?mode=server.

Server-Mode Only

The nodeset always runs server-mode under the ac-hmeqa env to keep litellm + PIL + numpy versions consistent with the other EQA stacks and to leave room for A-EQA's habitat-sim dependency. No work needed beyond setting HMEQA_PYTHON (already required for HM-EQA).

2. Canvas Nodes

What is not in this nodeset

• No step / done. EM-EQA is a single-pass DAG, not a loop. There is no agent action.
• No simulator camera. Frames come straight off disk — there's no cam_intr or pose to emit. (A-EQA, when added, will pull these from the Habitat env nodeset.)
• No multi-letter answer. Predictions are free-form text.

3. Composable Judge Pattern

The judge is not a single node — it is a 3-piece composition assembled in the graph:

        ┌─────────────────────────┐
reset.question  ─────────────►   │                         │
reset.answer_gt ─────────────►   │ llmCall (builtin)       │
reasoner.response ───────────►   │ template = mmbench.txt  │  ─response─►  parse_score  ─score_1to5─►  emit_metrics  ─metrics─►  outputPort
                                 │ profile = ""            │
                                 └─────────────────────────┘

Why the split:

• llmCall is generic — runs the OpenEQA mmbench.txt few-shot prompt against any LLM the user picks. Belongs to the framework's builtin catalog, not OpenEQA.
• parse_score is OpenEQA-specific — the 1–5 scale, the Your mark: N surface form, the fallback heuristics, and the -1 sentinel are conventions of this benchmark. Belongs to the OpenEQA nodeset.
• emit_metrics is OpenEQA-specific — the metric keys (openeqa_score, openeqa_llm_match) and the (s-1)/4 normalization are reporting conventions defined by the paper. Belongs to the OpenEQA nodeset.

This pattern generalises: any benchmark-specific LLM-judge can be expressed as llmCall + parse_X + emit_X_metrics where the parser and emitter live in the benchmark's own nodeset. See docs-site/docs/developer-guide/design-docs/graph-system.md §2.3 for the broader principle.

Inputs the judge llmCall needs

The judge llmCall is configured with three ports — question, answer, prediction — and the mmbench template (paper-faithful, copy-pasted from facebookresearch/open-eqa/prompts/mmbench.txt). Wire reset.question → judge.question, reset.answer_gt → judge.answer, reasoner.response → judge.prediction.

Outputs

• parse_score.score_1to5 — integer in [1, 5], or -1 if the response could not be parsed (no Your mark: N, no pure digit, no \b[1-5]\b fallback).
• emit_metrics.metrics — {"openeqa_score": <float>, "openeqa_llm_match": <float>}. Auto-harvested by BatchEvalRunner._collect_metrics and aggregated as a mean across episodes.
• emit_metrics.llm_match — (score − 1) / 4 in [0, 1] for parsable responses; 0.0 on sentinel.
• llmCall.response — the raw judge response (wire to a textViewer for audit / debugging).

Failure modes

The sentinel -1.0 survives mean aggregation — when the eval run completes you will see a non-physical mean if the judge silently failed on every episode. Inspect the judge llmCall.response per episode (via the wired textViewer) in the run JSONL to diagnose.

4. Configuration

env_openeqa_em__sample_frames config

env_openeqa_em__parse_score and env_openeqa_em__emit_metrics config

Both nodes are config-free — the score parsing rules and metric naming are fixed by OpenEQA convention. Drop them onto the canvas and wire them up.

Judge llmCall config (paper-faithful defaults)

The judge call is a builtin llmCall; the OpenEQA-specific recommendation is to set:

Manager-level config (passed to initialize(**kwargs))

5. Env panel Integration

OpenEQAEMEnvPanel (registered as env_openeqa_em on the canvas panel) cascades split → episode_index. OpenEQA's public release is val-only, so the split selector currently has a single option; the field is retained for forward-compatibility.

The env panel routes episode_index changes through mgr.set_episode_by_index, which drops the previous episode's frame cache so RSS stays bounded. The episode_reset signal clears any lifetime="episode" state containers downstream.

get_eval_metadata() returns step_budget=1 — EM-EQA is a single-pass DAG, not a loop. BatchEvalRunner uses this to pick a per-episode timeout of 1 × default_per_step_budget_sec (= 90 s today, judge-call dominated).

6. Dataset Layout

The loader expects this layout (override the root via OPENEQA_DATA_DIR):

data/openeqa/
    open-eqa-v0.json                              # list of question records (JSON)
    episodes/hm3d-v0/<episode_history>/
        00000-rgb.png, 00001-rgb.png, ...         # 1920×1080 PNG, lossless
    episodes/scannet-v0/<episode_history>/
        000000-rgb.png    (1296×968 RGB, lossless PNG)
        000000-depth.png  (640×480 uint16 depth, native ScanNet sensor res)
        000000.txt        (4×4 camera-to-world pose)
        ... (up to 600 frames per episode, capped at 20 sec @ 30 fps)
        intrinsic_color.txt  (4×4, fx≈1165.7, cx≈649.1 — for 1296×968 RGB)
        intrinsic_depth.txt  (4×4, fx≈575.7, cx≈320.3 — for 640×480 depth)
        extrinsic_color.txt, extrinsic_depth.txt  (4×4 extrinsics)

The loader is filename-schema agnostic — _list_frame_files accepts both flat layouts (frame_NNN.jpg) and the AIGeeksGroup <NNNNN>-rgb.png schema, and auto-filters basenames ending in -depth.* / -seg.* / etc. so non-RGB modalities never reach _load_frame_as_rgb.

Data source: AIGeeksGroup HuggingFace mirror

The upstream Meta-released frame tarball (open-eqa-hm3d-frames-v0.tgz on Dropbox, referenced in facebookresearch/open-eqa/data/README.md) is permanently expired — both the frames tarball and the matching states tarball used by the Habitat re-render path return Dropbox "Link Expired" pages as of 2026-05-04. The previously-used HF mirror Embodied1/open-eqa (parquet) was hard-capped at 32 frames per episode.

Our installer (scripts/data/fetch_dataset_openeqa.sh) now routes through AIGeeksGroup/OpenEQA — third-party packaging as per-episode .tar files, full original episode lengths, lossless PNG. Total ~87 GB (12 GB HM3D + 75 GB ScanNet). The installer downloads via huggingface_hub.snapshot_download, then extracts each tar with --strip-components=1 so files land directly under episodes/<subset>/<episode_id>/.

HM3D vs ScanNet provenance

The bonus ScanNet depth + pose channels are wasted by the current EM-EQA graph (which is RGB-only) but unblock future SpatialNav / 3D-grounded methods without a re-render.

Frame counts vs old mirror

The K=50 default in the multi-frame graph now actually pulls 50 frames on most HM3D episodes (was silently 32). For ScanNet, K=50 from a 600-frame episode under stride sampling matches the paper's gpt4v.py --num-frames=50 default.

Question-record schema (tolerant)

_normalize_question projects each upstream record onto a canonical schema with these field-name aliases:

If your upstream JSON uses different keys, edit _QUESTION_FIELD_ALIASES near the top of workspace/nodesets/env/env_openeqa_em.py.

Frame-directory layout

_list_frame_files searches data/openeqa/episodes/<episode_history>/ and the common subdirectories frames/, rgb/, color/, images/. The first non-empty match wins. Files are lex-sorted — name your frames so the natural sort is the temporal sort (e.g. 0001.jpg, 0002.jpg, …).

7. Example Graphs

The 6 baselines shipped in facebookresearch/open-eqa/openeqa/baselines/ collapse to three distinct inference logics, covered by three graphs. Switching vendor (OpenAI / Anthropic / Google / local LLaMA) is a profile-dropdown change inside each graph — no new graph required.

All three graphs are pure DAGs (step_budget: 1, no iter_in/iter_out). Hyperparameters in the configs (temperature=0.2, max_tokens=128, prompts) match the upstream prompts/*.txt and baseline --num-frames defaults; the judge llmCall defaults to temp=0.2, max_tokens=128 (paper used 32 with gpt-4-1106-preview — modern chat models need more headroom).

Faithfulness gaps: - llmCall does not expose seed. The paper sets seed=1234 for OpenAI calls; our runs are non-deterministic. Mean LLM-Match across ≥30 episodes is the right comparison, not single-episode scores.

7.1 openeqa_em_multiframe.json — Group B (multi-frame VLM)

Reproduces gpt4v.py (K=50, paper-faithful — full episodes available via AIGeeksGroup mirror). Topology:

[reset] ── frames ──► [sample_frames k=50] ── sampled ──► [vlmCall (rgb+question)] ── response ──┐
                                                                                                   │
[reset] ── question ──────────────────────────────────────────────────────────────► [judge llmCall (mmbench)]
[reset] ── answer_gt ─────────────────────────────────────────────────────────────►        │
                                                                                            │
                                                                                            ▼
                                                                              [parse_score] ──► [emit_metrics] ──► [outputPort]

llmCall auto-detects VLM mode at runtime from the rgb input being a non-empty image list — no vlm config flag is needed.

7.2 openeqa_em_single_frame.json — Group C (single-frame VLM, ablation)

Sits between blind-LLM (no vision) and multi-frame (K=50) on the OpenEQA baseline ladder — the standard ablation that isolates "first-look" vision contribution from multi-frame fusion. K=1 with strategy="first_last_middle" returns the middle frame. Topology is identical to multi-frame, only the sample_frames config changes.

7.3 openeqa_em_blind_llm.json — Group A (language-only)

Reproduces gpt4.py / llama.py / gemini-pro.py. Frames are not loaded — the model answers from prior knowledge. Topology:

[reset] ── question ──► [llmCall (TEXT, blind-llm prompt)] ── response ──┐
                                                                          │
[reset] ── question ──────────────────────────────────────────► [judge llmCall (mmbench)]
[reset] ── answer_gt ─────────────────────────────────────────►        │
                                                                        │
                                                                        ▼
                                                          [parse_score] ──► [emit_metrics] ──► [outputPort]

The blind-llm.txt prompt is split between the reasoner llmCall's system_prompt (few-shot examples) and template (Q: {question}).

Running any of the three graphs

1. bash scripts/install/install_ac_hmeqa.sh (one-time, brings up the ac-hmeqa conda env)
2. bash scripts/data/fetch_dataset_openeqa.sh (downloads HF mirror + materialises 32 frames/episode under data/openeqa/)
3. Make sure your active LLM profile (/api/profiles/) points to a model that can handle the role — vision-capable for graphs B/C, any chat model for A.
4. Open the graph and hit Play, or batch-eval via POST /api/eval/v2/start with episode_count=K, worker_count=N (parallelism="replicated" spawns N tagged subprocesses, each loading the JSON + its own episode-frame cache).

Profile note: All three graphs ship with llmCall.profile = "", which resolves to the user's active profile at run time. If you want different LLMs for the reasoner vs the judge, override profile per-node in the canvas UI; do not edit the saved graph JSON to bake in a profile name (graph-system.md §2.3).

8. Environment Setup

# 1. Bring up the hmeqa conda env (shared with HM-EQA)
bash scripts/install/install_ac_hmeqa.sh

# 2. Stage the OpenEQA data (AIGeeksGroup HF mirror; ~87 GB total: 12 GB HM3D + 75 GB ScanNet)
#    Use --filter hm3d or --filter scannet to fetch only one subset.
bash scripts/data/fetch_dataset_openeqa.sh

# 3. Tell the framework which interpreter hosts this nodeset
export HMEQA_PYTHON=/home/$(whoami)/miniforge3/envs/ac-hmeqa/bin/python

# 4. Make sure your active LLM profile points at a model with credit
#    (graphs default to `llmCall.profile = ""` which resolves to active).
#    Use the Profiles panel, or POST /api/profiles/activate, to switch.

Override the data root if you keep OpenEQA outside the repo:

export OPENEQA_DATA_DIR=/mnt/datasets/openeqa

9. License

OpenEQA is released by Meta AI under a research license documented in the upstream facebookresearch/open-eqa repository. Confirm the license terms before redistributing any data or eval outputs.