EQA Support Status

This page does two jobs. Part I — Support answers which EQA methods AgentCanvas runs or wants to run: a single support matrix (verified · in-progress · planned), then the env / reasoner / method machinery underneath. Part II — Landscape places those methods in the wider EQA field — representative & most-cited reasoners and the 2024–26 frontier (§9) — condensed from the EQA — Methods survey. Companion to VLN Support Status and VLA Support Status.

Part I — Methods we support

1. Support matrix

Every EQA graph AgentCanvas ships, is building, or has planned (§8). This is the single source of truth for "what we support" — the env / reasoner / method backing is in §2–§4, planned adoptions in §8, the wider field is Part II. Verified graphs live under eqa/verified/, the rest under eqa/unverified/.

All three OpenEQA EM-EQA graphs reproduced on 2026-05-04. Frozen artefacts (summary + graph + log.jsonl.gz with raw VLM and judge text) live under outputs/archive/openeqa_em_baselines_2026-05-04/ — the README in that folder is the canonical reproduction record. Headline ordering matches paper expectation: multi-frame ≫ blind > single-arbitrary-frame.

We outperform paper baselines because the reasoner + judge are gpt-4o (2025); the paper used gpt-4 / gpt-4-vision-preview (2024). The +24 pp gap on multi-frame is dominated by the model upgrade.

VLN graphs are out of scope (see VLN Support Status); VLA graphs are out of scope (see VLA Support Status).

2. Environment Nodesets

EQA environments wrapped as BaseNodeSets under server mode. Both share the ac-hmeqa conda env in server mode (parallelism="replicated").

2.1 HM-EQA (E6, in progress)

• Files: workspace/nodesets/env/env_hmeqa/__init__.py + hmeqa_renderer.py + hmeqa_replay.py
• Source: House-Mesh EQA over HM3D — multi-choice questions on semantic scenes
• Capabilities: 5 tools incl. free-pose teleport
• Method side: paired with explore_eqa.py + explore_eqa_tsdf.py (see §3)
• Doc: HM-EQA nodeset
• Constraint discovered during integration (TODO #17): live TSDFPlanner objects don't cross the server-mode JSON IPC boundary (numba-JIT'd volumes aren't pickle-friendly). The explore-eqa nodeset works around this with subprocess-local module globals keyed by episode_id, not state containers — load-bearing design constraint for future container work

2.2 OpenEQA (E9, shipped 2026-05-04)

• File: workspace/nodesets/env/env_openeqa_em.py
• Source: OpenEQA EM-EQA (24.06) — free-form Embodied QA with LLM-as-judge (1–5 → LLM-Match)
• 5 tools: reset / episode_info / sample_frames / parse_score / emit_metrics — judge LLM call uses framework builtin llmCall; OpenEQA-specific score parsing + metric naming live in two thin nodeset nodes
• Dataset path (the install actually works today): HuggingFace mirror Embodied1/open-eqa (32 frames/episode parquet) — installer rewritten to use this since the upstream Meta Dropbox tarballs are permanently expired (scripts/data/fetch_dataset_openeqa.sh)
• Critical config — image_size (default 512): sample_frames long-side resize. Mirrors paper baseline (openai_utils.py:30); the default keeps a 32-frame batch under provider single-request size caps (OpenAI rejects >50 MB; raw 1920×1080 PNGs total ~70 MB and silently fail otherwise — that bug was the reason early multi-frame smokes scored 1/5 across the board)
• Doc: OpenEQA nodeset

2.3 OpenEQA A-EQA (E10, deferred — blocked on data)

Active-EQA mode would reuse HM-EQA's Habitat manager + an explore-eqa-style termination policy on top of the E9 nodeset, plus a free-text answer-emit action. Not yet shipped — and the blocker is data, not method:

• The original OpenEQA Meta-Dropbox tarballs (full-fidelity HM3D / ScanNet frames + the states archive used for Habitat re-rendering) are permanently expired — verified 2026-05-04 (Link Expired HTML response, see the rationale comment block at the top of scripts/data/fetch_dataset_openeqa.sh).
• The HuggingFace mirror Embodied1/open-eqa is enough for EM-EQA (32 frames per episode, parquet) but does not carry the scene meshes / state artefacts an active explorer needs.
• Method-side dependencies (answer-emit verb on a unified action contract — TODO #46) are real but secondary; even if those landed today, A-EQA would still be blocked until we either (a) re-render via Habitat from a fresh path-fix patch + recovered states, or (b) find a community mirror of the original tarballs.

3. Reasoner / Method Nodesets

EQA-shaped graphs are reasoner fragments, not full agents — pure (frames, question) → (answer, score) functions. They wrap as composite graph_nodes; default to no iter_in / iter_out (per project memory eqa_reasoner_shape).

3.1 Explore-EQA (paper port)

• File: workspace/nodesets/explore_eqa.py
• Source: Explore-EQA — agent that explores HM3D scenes, scores VLM token-likelihoods over frontier choices, and decides when to answer
• Pairs with: HM-EQA env nodeset (§2.1) — example graph explore_eqa_hmeqa.json
• Backend lock-in (v1): Prismatic-locked. ExploreEQANodeSet.initialize() loads Prismatic; vlm.get_loss() is called directly from VLMScoreStep.forward() — same pattern as the retired policy_cma port. Decoupling tracked as TODO #50
• Doc: ExploreEQA nodeset

3.2 Explore-EQA TSDF utilities

• File: workspace/nodesets/explore_eqa_tsdf.py
• Source: vendored TSDFPlanner + geom utilities from explore-eqa, kept verbatim with minor tweaks
• Why a separate file: TSDFPlanner state is what hits the pickle/JSON-IPC wall mentioned in §2.1 — keeping it in its own module makes the subprocess-local-globals pattern easier to reason about

3.3 OpenEQA reasoner nodes

• File: same workspace/nodesets/env/env_openeqa_em.py as the env (the OpenEQA nodeset bundles both env-side reset/sample/info and judge-side parse/emit). The judge LLM call itself is the framework built-in llmCall — by design, so users can swap providers without touching this nodeset (project memory: llmCall.profile = "" so the active profile resolves at run time)
• Judge model is gpt-4o for all variants — gpt-4-1106-preview referenced in the paper has been retired by OpenAI (per project memory feedback_openeqa_reasoner_model); don't pin the older model in new graphs.

3.4 ToolEQA (redone 2026-06-08 — monolith, runs e2e)

• Status: rebuilt monolith-first after the unverified first port was removed. tooleqa__step (backend, transformers.agents ReAct in-process) dispatches over server-mode HTTP to Qwen2.5-VL-3B (vlm_qwen2_5_vl, LLM + VQA), DetAny3D (model_detany3d, 2D/3D detect), and a new fused tooleqa_explore__go_next that runs the full Explore-EQA TSDF frontier step + teleport (reusing _explore_eqa_tsdf.TSDFPlanner; the first port wrongly treated go_next as discrete dead-reckoning). Graph workspace/graphs/eqa/unverified/tooleqa_hmeqa.json is a Design-B loop — env_hmeqa__step is not a loop node (go_next steps it internally), and the step self-evaluates to graphOut. Housekeeping 2026-07-04: the monolith had been shadowed by a same-named root-level copy while the organized slot held the removed first-port remnant — root copy deleted, monolith now lives at the standard path. End-to-end smoke worked pre-refactor (ep0 → correct 'B'), but the graph currently starves at iteration 0 (iterIn loop-carry never seeded — post 2026-06-11 two-sided-iterOut semantics) and is in the unverified re-eval wave (roadmap #63, #64); SR aggregate + 3B convergence also still being tuned. Then decompose into a clean multi-node graph (voxposer monolithic → decomposed path).
• Retained for the redo: model_detany3d server nodeset + data/detany3d/ weights + ac-detany3d conda env + install_ac_detany3d.sh; the tooleqa-hmeqa eval profile; the upstream fetch stub workspace/nodesets/_upstream/tooleqa/fetch_upstream.sh; the 2026-05-10 port plan (.claude/plan/) as porting reference.
• PortBench v1 scope (unchanged): HM-EQA only — reuses the explore_eqa_hmeqa.json substrate; OpenEQA-A-EQA (data-blocked, see §2.3) and EXPRESS-Bench (high integration cost) are explicitly out of scope for v1. Decision recorded in project memory project_tooleqa_hmeqa_only.

4. Ready-to-Run Graphs

tooleqa_hmeqa.json was redone monolith-first on 2026-06-08 and runs end-to-end again — see §3.4.

When forking a new EQA experiment, start from openeqa_em_multiframe.json — it's the strongest baseline and the configuration most closely matching the OpenEQA paper. For language-only (text-baseline) ablations, start from openeqa_em_blind_llm.json. The single-frame graph is an ablation slot, useful only as a sanity check that single-arbitrary-frame ≈ blind (it does — single-frame underperforms blind, in line with paper expectations that one random frame adds noise rather than signal).

Frozen reproduction package: outputs/archive/openeqa_em_baselines_2026-05-04/ — contains the three summary.json + graph.json + log.jsonl.gz (raw VLM answers + judge "Your mark: N" text), plus a README with reproduction commands.

5. Datasets, Question Format, and Metrics

5.1 HM-EQA

• Dataset: HM3D semantic scenes; HF mirror bingCS/hm3d — 264/266 scenes downloaded as of 2026-04-30; 24 missing only navmesh (env auto-recomputes); 454/500 questions full-fidelity
• Question format: multi-choice
• Scoring: Prismatic VLM token-likelihood over choice tokens (vlm.get_loss)

5.2 OpenEQA EM-EQA

• Dataset: HF mirror Embodied1/open-eqa — 32 frames per episode in parquet form. HM3D filter: 557 questions / 63 scenes; ScanNet filter: 1079 questions (untested in our pipeline). Full benchmark in the original paper is 1636 questions.
• Question format: free-form (open-ended)
• Scoring: LLM-as-judge — judge returns 1–5; the 1–5 → LLM-Match conversion is the canonical OpenEQA metric. The judge LLM is configurable via the framework's llmCall profile, not pinned to a specific provider
• Frame size: long-side 512 px PNG base64 (matches paper openai_utils.py:30). Configurable via sample_frames.image_size; don't disable unless you've verified your VLM provider accepts >50 MB per request.

5.3 Why two scoring paths matter architecturally

EQA is the first benchmark family in the repo that requires two distinct scoring shapes:

• Multi-choice / VLM token-likelihood (HM-EQA, Prismatic) — one library call, deterministic
• Free-form / external judge (OpenEQA, any LLM) — separate inference call, prompt-engineered

Both are wired through the same canvas pattern: env nodeset emits frames + question; reasoner emits answer + per-choice or free-text payload; a thin "scoring" subgraph emits the metric. This is what made OpenEQA E9 a useful generalisation exercise.

6. Eval Infrastructure

6.1 BatchEvalRunner + EnvWorkerPool (ADR-eval-002)

Same shared pipeline VLN and VLA use (agentcanvas/backend/app/agent_loop/eval_batch.py + env_worker_pool.py):

• One fresh LoopRunner per episode
• worker_count > 1 spawns N tagged env subprocesses; per-worker env_panel_overrides + server_url_overrides
• Per-episode timeout: max_steps × per_step_budget_sec — for EQA you typically want a generous budget because of judge-LLM latency

6.2 Parallelism posture (ADR-server-003)

• HM-EQA and OpenEQA env nodesets stay parallelism="replicated" — they're stateful and thread-affine; N tagged copies under worker_count > 1
• The judge LLM uses the framework's llmCall and inherits whatever profile / batching the runtime is configured for

6.3 Reasoner-shape consequence

Because EQA reasoners are pure (frames, question) → (answer, score) functions (no iter_in / iter_out by default — project memory eqa_reasoner_shape), they fit naturally into a per-episode one-shot evaluation rather than per-step VLN-style streaming. Plan eval graphs accordingly: one or a few reasoning steps per episode, terminating cleanly so BatchEvalRunner can advance.

7. Known Gaps

EQA-relevant items in roadmap.md:

8. Planned Adoptions — Method & Benchmark Tracks

A frank survey of EQA papers from the last ~12 months, scored for AgentCanvas integration. Two tracks: Method (agent / reasoning approaches we'd port as graph + nodeset) and Benchmark (datasets / metrics we'd add to eval infrastructure). Citations are 2026-05-05 snapshots (Google Scholar primary, Semantic Scholar fallback; numbers <5 are noisy and labelled as such).

8.1 Track A — Method

8.2 Track B — Benchmark / Eval Infrastructure

8.3 Landing order

Two tracks are interleaved so the benchmark upgrade lands first; every subsequent method port consumes the new metric.

After items 1-4 land, paper #2 has N=3+ case studies (existing OpenEQA three + ToolEQA + GraphPad), the cross-framework-porting benchmark is structurally complete; paper #1 has both an EAC reward signal and a strong scene-graph baseline to evolve against.

Part II — Field landscape

9. Landscape — representative methods & frontier

Where the supported EQA subset sits in the wider field, condensed from the research survey EQA — Methods (cite counts are that page's Semantic Scholar snapshot). Ours maps each method to our port status: ✅ shipped · 🔁 in-progress · 🅼 planned (§8) · — not ported.

9.1 Representative & most-cited

9.2 Frontier (2024–2026)

Our supported set spans the field's two live archetypes: EM-EQA reasoners (OpenEQA baselines — episodic-memory VLM judging) and active exploration (Explore-EQA), with tool-call (ToolEQA) in progress and the scene-graph line (GraphEQA / GraphPad) planned (§8). The 2025–26 frontier is dominated by memory-centric and Gaussian-splat spatial-memory variants — none ported yet.