ExploreEQA on AgentCanvas — paper analysis + explore_eqa port

Explore until Confident (Ren et al. 2024; repo Stanford-ILIAD/explore-eqa) is a zero-shot embodied-QA agent: a Prismatic VLM scores token likelihoods — answer letters A/B/C/D and a Yes/No relevancy signal — every step, a semantic-value TSDF turns VLM "which direction would you explore?" answers into frontier weights, and a post-hoc weighted vote over the whole trajectory emits the final letter. This page documents the 8-node explore_eqa method nodeset plus the vendored TSDFPlanner driving env_hmeqa in explore_eqa_hmeqa.json, with all four per-step VLM calls decoupled onto vlm_prismatic__score_tokens wires.

1. Upstream method analysis (Ren et al. 2024)

Upstream is a single rollout script, run_vlm_exp.py, around a TSDFPlanner (src/tsdf.py) and a Prismatic VLM exposing get_loss(image, prompt, tokens) → probs. Per episode it formats the question, derives a scene-size step budget, and loops:

1. Question formatting (:84–89): vlm_question = question + "\nA. <choice>" … for the fixed candidates A/B/C/D.
2. Budget + TSDF init (:132–145): num_step = int(√scene_size · max_step_room_size_ratio) (ratio 3 in cfg/vlm_exp.yaml); TSDFPlanner(vol_bnds, voxel_size=0.1, init_clearance=0.5·2).
3. Black-frame gate (:179–182): if black pixels ≥ 0.5·W·H (robot outside the floor mesh) the entire scoring + mapping block is skipped and the step is recorded as pred = uniform(4), rel = [0.01, 0.99] (:306–309) — a near-zero-weight placeholder in the final vote.
4. TSDF fusion (:185–193): integrate(rgb, depth, cam_intr, cam_pose_tsdf, obs_weight=1.0, margin_h=0.6·H, margin_w=0.25·W).
5. Answer + relevancy scoring (:197–211): pred = get_loss(rgb, vlm_question + "\nAnswer with the option's letter from the given choices directly.", [A,B,C,D]); rel = get_loss(rgb, "…Are you confident about answering the question with the current view? Answer with Yes or No.", [Yes,No]).
6. Frontier visual prompting (:213–263): find_prompt_points_within_view returns candidate pixels; if at least min_num_prompt_points (2) are found, grey circles (radius 18, fill (200,200,200), outline width 3) with black letters (OpenSans 30, anchor mm) are drawn on a copy of the view.
7. LSV + GSV (:269–294): LSV = get_loss(annotated, "…Which direction (black letters on the image) would you explore then? Answer with a single letter.", letters[:n]) · n/3; GSV = exp(P(Yes)/gsv_T)/gsv_F with gsv_T=0.5, gsv_F=3, asked on the unannotated image. sv = lsv·gsv is written into the value volume via integrate_sem(sv, radius=1.0) (:297–301).
8. Next pose (:312–321): find_next_pose(pts, angle, flag_no_val_weight = step < min_random_init_steps, **cfg.planner) — frontier-weighted sampling (unexplored-rate · unoccupied-rate · exp(val/T) weights, Gaussian-smoothed value map); the first 2 steps ignore value weights (random-walk init). The agent is teleported (agent.set_state).
9. Post-hoc vote (:337–354): weighted matrix rows P(Yes)_t · pred_t; success_weighted = argmax of the per-letter max over steps; success_max = argmax of the weighted row at the highest-relevancy step. Both variants are reported.

1.1 Upstream invariants worth marking

• Everything is token likelihood, never generation. All four calls go through get_loss(image, prompt, tokens) — letters, Yes/No. There is no free-form decoding anywhere in the loop.
• The black frame is an outside-the-floor sentinel. It skips the whole block — no fusion, no scoring, no frontier work — and the step enters the vote as pred=uniform, rel=[0.01,0.99], i.e. with ~1% weight (:306–309).
• Answering is free. The final letter comes from a post-hoc vote over stored per-step probs (:337–354); no extra exploration or VLM call happens at episode end.
• The yaml overrides the code. tsdf.py function defaults differ from cfg/vlm_exp.yaml (e.g. unexplored_T 0.5 vs 0.2); upstream always passes **cfg.planner / **cfg.visual_prompt, so the yaml values are what actually ran.
• The step budget is scene-derived: int(√scene_size · 3) — small rooms get short episodes. The paper's conformal-prediction stopping ("until confident") is not in the released code; the rollout always runs the full budget.
• init_clearance is doubled at the call site (:144): the yaml says 0.5, the planner receives 1.0.

2. The AgentCanvas port

The port splits the rollout into 8 reasoning/mapping nodes and moves every VLM call onto wires: each of the four get_loss call sites becomes a vlm_prismatic__score_tokens canvas-node instance sitting between a pre node (emits image / prompt / tokens) and a post node (consumes probs). The nodeset runs server-mode in the ac-hmeqa conda env, co-located with env_hmeqa, because the vendored TSDFPlanner holds numba-JIT'd volumes that cannot cross JSON IPC. Movement is the upstream teleport, executed env-side by env_hmeqa__step_pose.

2.1 Per-step flow

Drawn simplified in four places. ① The pre/post pairs are collapsed: vlm_score_pre fans the raw view into two score_tokens instances (pred and rel) whose probs land in vlm_score_post, which appends them to _SCORE_HISTORY[episode_id]; frontier_pre does the same for LSV (annotated image) and GSV (base image) into frontier_post. ② The loop carry is heavy, the opposite of SmartWay: iter_out persists rgb / depth / cam_pose / pose_normal / angle / step_index / episode_id / choices / answer_gt from loop_obs (a second observe_egocentric triggered by env_step.info); run-invariants (question / cam_intr / tsdf_bnds …) ride iter_in init ports (persist=true). ③ The loop ends at the env’s per-scene exploration horizon: env_step sets truncated=true when step_index ≥ num_step, wired as the single iter_out.stop feeder (two-pivot model, no separate termination node). Rebuild note (2026-06-13): three loop bugs from the iterIn+gym refactors were fixed — the body’s observation inputs are now dual-wired (init_* for iteration 0 + iterout_* after, init ports persist=false); iter_out’s five loop_obs-sourced carry ports are marked required so the boundary waits for the end-of-body observation instead of firing early on env_step’s partial output; and the num_step horizon stop (lost when the gym migration split the old done into error-only terminated) was restored via truncated. ④ Post-loop, iter_out.final_episode_id / final_choices / final_answer_gt fire aggregate_answer, whose pred_letter then fires evaluate — a pivot-rooted cascade, which is the sound shape (contrast the final_fire incident in §4). One wart: next_pose.step_index is double-wired from both iter_in.iterout_step_index and iter_in.step (§3E).

2.2 Node inventory

2.3 State & memory

This nodeset deliberately does not use graph_state containers: TSDFPlanner holds numba-JIT'd volumes and lambda-captured state that cannot cross the JSON IPC boundary containers use in server mode. The on-wire episode_id plays the container-key role instead. Entries are retained across episodes — a recorded race fix (2026-05-03): a global .clear() in the planner builder used to wipe sibling episodes' history under parallelism="shared" with worker_count > 1. Bounded growth ≈ 30 MB × ≤50 episodes per run, dropped at subprocess teardown.

2.4 Boundary contract

Three subprocesses, all under the ac-hmeqa env ($HMEQA_PYTHON): env_hmeqa (habitat-sim, ×N workers), explore_eqa (this nodeset, singleton, co-located TSDF), vlm_prismatic (Prismatic VLM, singleton). Env verbs consumed: reset (metadata incl. tsdf_bnds and the per-episode budget surfaced through the framework's step_budget resolver — num_step = int(√scene_size·3) computed env-side, env_hmeqa/__init__.py:466), seed + loop observe_egocentric, step_pose (teleport; env appends floor_height and converts normal→habitat frame), evaluate. Foundation-model boundary: method nodes never load a VLM — any nodeset exposing score_tokens(image, prompt, tokens) → probs can replace vlm_prismatic; its degraded mode (uniform probs when the model can't load) keeps the graph structurally runnable without a GPU.

2.5 Prompt assets

All four prompts are inline f-strings in explore_eqa.py, byte-checked against upstream this audit: pred (run_vlm_exp.py:197–200), rel (:208 — split across a Python string concatenation in the port, rendered output identical), LSV (:270), GSV (:285). Drawing constants _DRAW_LETTERS / _CIRCLE_RADIUS=18 / _FONT_SIZE=30 mirror :235–263 + cfg.visual_prompt.circle_radius, including upstream's odd no-op font_size=12 argument alongside the size-30 font object.

3. Delta vs upstream — four buckets

A. Preserved verbatim

B. Forced by substrate (script → typed graph)

C. Forced by environment / cost

D. Intentional divergences

E. Unexplained / defects

Probed and cleared: rel prompt byte-equality across the port's string split; VLMScorePost empty-probs fallback reproducing upstream's exact placeholder values; vote formulas against :337–354 (incl. the subtle per-letter max-pool); init_clearance·2; drawing constants incl. the dead font_size=12; vendored find_next_pose / find_prompt_points_within_view semantic diff (matplotlib stripping + yaml-pinned defaults only); integrate_sem body identical. No port equivalence test exists — eval parity (§4) is the only executable check.

4. Evaluation

Incident history (the final_fire lesson): commit bc6aa8f (2026-05-11) set final_fire=true on env_hmeqa__evaluate. The executor's _final_fire_pass re-fires flagged nodes alone, without their upstream nodes — so evaluate re-fired with pred_letter=None and SR collapsed 0.43 → 0.0 silently for 9 days (2026-05-11 → 2026-05-20). Fixed by removing the flag. The current graph uses the sound shape instead: the post-loop chain is rooted at the iterOut pivot (final_* ports → aggregate_answer → evaluate), so every link re-fires with real inputs.

Current status: explore_eqa_hmeqa.json was promoted to workspace/graphs/eqa/verified/ on 2026-06-13 after the loop rebuild + confirm run (20260613_151701, 50 ep, SR 0.42). The graph had been silently loop-broken since the iterIn + gym refactors (marked verified then but never re-run); the rebuild restored a correctly-iterating loop that harvests a verdict on every episode and stops at the scene-natural horizon (§2.1 rebuild note). The smartway-class TODO #64 harvest miss does not apply (evaluate is post-loop, not step-triggered), now confirmed by the clean harvest. Profile caveat: perf_explore_eqa_hmeqa’s declared 10 workers crashed at cold start (10 simultaneous habitat GL contexts); 5 workers ran clean — the 10-worker calibration needs revisiting.

5. Usage

# load — method + env + foundation model (all server-mode, hmeqa env)
POST /api/components/nodesets/explore_eqa/load?mode=server
POST /api/components/nodesets/env_hmeqa/load?mode=server
POST /api/components/nodesets/vlm_prismatic/load?mode=server

# graph
workspace/graphs/eqa/verified/explore_eqa_hmeqa.json

# batch eval — graph-only form: /experiment:run <profile> <graph_name> [key=value ...]
/experiment:run perf_explore_eqa_hmeqa explore_eqa_hmeqa          # 100 ep HM-EQA val
/experiment:run smoke_explore_eqa_hmeqa explore_eqa_hmeqa episode_count=3

6. What this nodeset is NOT

• Not "until confident". The paper's conformal-prediction calibrated stopping is not in the upstream release either; both upstream and the port run the scene-derived fixed budget. The name describes the paper, not the shipped code.
• Not a Prismatic wrapper. Method nodes emit (image, prompt, tokens) bundles; any score_tokens provider (LLaVA, Qwen-VL, …) slots in without touching method code.
• Not env logic. Teleport execution, floor-height handling, scene bounds, and the per-episode budget all live in env_hmeqa.
• Not an answer action. The letter comes from the post-hoc vote; nothing is "spoken" during the episode (faithful to upstream).

7. Source files