EXPRESS-Bench NodeSet
The EXPRESS-Bench NodeSet (EnvExpressNodeSet, workspace/nodesets/env/env_express/) wraps habitat-sim 0.3.x for exploration-aware open-vocabulary EQA on HM3D, implementing EXPRESS-Bench (25.03, ICCV 2025): 2,044 questionβtrajectory pairs over 777 GT exploration trajectories in 174 scenes. The agent explores by free-pose teleport hops (navmesh-snapped, geodesic path length accumulated), answers the question, and is scored by the benchmark's own gpt-4o-mini judge into the C / C* / E_path / d_T metric suite β grounding and correctness judged jointly, path efficiency and goal distance folded in on the env side.
Code descendant of HM-EQA's explore-eqa lineage β same simulator generation, same camera conventions (512Γ512 RGB-D, HFOV 90Β°, tilt 0), and the same ac-hmeqa conda env ($EXPRESS_PYTHON overrides). It is a separate nodeset rather than a dataset spec inside env_hmeqa because the task shape differs: open-vocab answers with an LLM judge band instead of multiple-choice letter match, and per-episode step budgets that scale with scene size.
1. Overview
Benchmark identity
| Fact | Value |
|---|---|
| Paper | Beyond the Destination (EXPRESS-Bench) (25.03), Jiang et al., ICCV 2025, SYSU HCP |
| Questions | 2,044 open-vocab QA / 777 trajectories / 174 scenes (140 train + 34 val; both splits are eval-only for zero-shot methods β "train" is the HM3D scene split, not training data) |
| Judge | gpt-4o-mini with the benchmark's evaluation.txt prompt; returns "Ξ΄, Ο" β grounding Ο first, correctness Ξ΄ second |
| Metrics | C (answer correctness Γ grounding), C* (grounding-only ceiling), E_path (path-efficiency-weighted C), d_T (final distance to GT goal; inf episodes skipped upstream β mirrored as d_t=-1 + d_t_valid=0) |
| Paper reference rows | Fine-EQA C 40.55 / C* 63.95 / E_path 16.22 / d_T 6.43 Β· RE (random) C 36.95 / C* 62.75 / E_path 12.06 Β· FBE C 38.60 Β· human C* 83.99 |
Architecture
The three-layer shape shared by every env nodeset:
ExpressEnvManager(singleton) β owns one habitat-sim simulator, a lock, and a single-thread executor pinning simulator work to one OS thread. Scenes load as bare.basis.glb(upstream'ssemantic_sensorflag is dead code β itsmake_simple_cfgbuilds color+depth only; mirrored deliberately).- Canvas nodes β nine: six gym verbs plus the FBE frontier sampler and the two baseline-loop routing nodes, all thin adapters over the manager (Β§2).
EnvExpressNodeSetβserver_python = conda_env_python("ac-hmeqa", "EXPRESS_PYTHON"), auto-hosted server-mode with the same NVIDIA driver-570 EGLLD_PRELOADshim as HM-EQA.default_per_step_budget_sec = 30.0(LLM-in-the-loop sizing, same rationale as ObjectNav's).
2. Canvas nodes
| Node type | Display name | Input ports | Output ports | Purpose |
|---|---|---|---|---|
env_express__reset |
EXPRESS: Reset | trigger (ANY, optional) |
question, answer, scene, episode_id, trajectory_id, question_type, num_step, floor_height, tsdf_bnds, geodesic_distance |
Begin episode β metadata only, no observation. answer is the GT free-form answer (judge input; never show the agent). num_step is the per-episode, scene-size-dependent budget. tsdf_bnds hands TSDF-frame volume bounds to mapping methods. |
env_express__step_pose |
EXPRESS: Step (pose teleport) | action (TEXT, free-pose JSON) |
reward, terminated, truncated, info, step_index, episode_id |
Teleport to a waypoint: navmesh snap_point with NaN β get_random_navigable_point_near(last, 3) fallback, geodesic hop distance accumulated into path_len (drives E_path) β mirrors upstream main.py:407-415. Unparseable action JSON or manager error sets terminated=truncated=True (designed loop-stop). truncated=True also once step_index reaches num_step. |
env_express__sample_waypoint |
EXPRESS: Sample Waypoint (random) | trigger (ANY) |
action (TEXT) |
Random-exploration action source (the paper's RE baseline hop): navigable point within 3 m of the agent β the same get_random_navigable_point_near call as upstream β plus a uniform random yaw from a seed-derived RNG, emitted as step_pose's free-pose JSON. |
env_express__sample_frontier |
EXPRESS: Sample Frontier (FBE) | trigger (ANY) |
action (TEXT) |
Frontier-exploration action source (the paper's FBE baseline): TSDF-fuses the current depth frame (TSDFPlanner.integrate, skipped on black frames) then picks the next frontier pose (find_next_pose with the fine_eqa.yaml planner block), emitted as step_pose's free-pose JSON. Planner state is vendored verbatim (_tsdf.py / _geom.py from upstream src/) and rebuilt per episode; regs_list=[] keeps the GOE branch permanently off. Planner failure falls back to the RE random hop. |
env_express__stop_gate |
EXPRESS: Stop Gate | rgb, stop_text |
continue, stop, rgb_final, stopped_early |
Routes the per-step early-stop decision (upstream main.py:200-210): "yes" from the stop LLM on a non-black frame emits stop + the decision frame to iterOut; otherwise continue triggers the next hop. Black frames never stop β upstream skips the check entirely on black frames. |
env_express__answer_router |
EXPRESS: Answer Router | question, stopped_early, rgb |
question_stop, rgb_stop, question_trunc |
Post-loop branch (upstream main.py:204-210 vs 419-422): early stop β answer.txt prompt WITH the decision frame; budget truncation β random_answer.txt prompt WITHOUT any image. Emits exactly one branch, so exactly one of the two answer llmCalls fires. |
env_express__observe_egocentric |
EXPRESS: Observe (egocentric) | trigger (ANY, optional) |
rgb, depth, pose_normal, angle, intrinsics, cam_pose_tsdf, β¦ |
Idempotent pull of the current first-person frame: 512Γ512 RGB-D, normal-frame position + yaw, camera intrinsics, and the TSDF-frame extrinsic that upstream's mapping stack consumes. |
env_express__judge_prompt |
EXPRESS: Judge prompt | pred_answer (TEXT) |
user, system |
Assemble the benchmark's evaluation.txt judge prompt (question + GT answer + model response) exactly like upstream gpt.py:prompt_make. Wire user plus the final frame into a gpt-4o-mini llmCall. |
env_express__evaluate |
EXPRESS: Evaluate | pred_answer, judge_text |
metrics |
Parse "Ξ΄, Ο" (grounding first) mirroring upstream evaluation.py and fold path efficiency + goal distance into c, c_star, e_path, d_t (+d_t_valid, delta, sigma, judge_ok, path_len, gt_geodesic, steps_taken, num_steps). Filter aggregate d_T on d_t_valid. |
Action contract
step_pose consumes free-pose JSON β the last line of a planner LLM's reply, or sample_waypoint's output, verbatim:
{"position_normal": [x, y], "angle": yaw_radians}Positions are normal-frame 2D (floor height appended env-side); the waypoint is navmesh-snapped, so methods may propose freely. Hops are conventionally capped at 3 m by prompt, mirroring upstream's planner limit β the env does not enforce it.
3. Env panel
ExpressEnvPanel owns episode placement: split (val / train / all) β episode_index. Episode ids count from array order β val episodes sit at ids 1635+, so a "val episode 0" sanity check must select by index within the split, not by id.
4. Data staging
| Path | Contents |
|---|---|
data/hm3d/express_bench/express-bench.json | The 2,044-question release |
data/hm3d/hm3dsem/ | Flat HM3D basis-scene pool β all 174 EXPRESS scenes (and all 828 MT-HM3D scenes) audited present with non-empty glb + navmesh; val scenes symlinked from scene_datasets/hm3d/val, train scenes fetched from the bingCS/hm3d HF mirror with official-tar backfill |
workspace/nodesets/env/env_express/prompts/evaluation.txt | Vendored judge prompt (benchmark-native) |
5. Graphs: express_probe, express_re, express_fbe
All in workspace/graphs/eqa/unverified/. The two baseline graphs are step-for-step faithful to upstream main.py (2026-08-14 rework): each iteration first runs the gpt-4o-mini stop check (explore.txt verbatim) on the current frame β starting with the seed frame, matching upstream's cnt_step=0 β then stop_gate either exits the loop or triggers the exploration hop. The verdict band branches on how the loop ended: early stop β answer.txt WITH the decision frame; budget truncation β random_answer.txt WITHOUT any image (main.py:421). Judge = evaluation.txt + final frame β Ξ΄, Ο β C / C* / E_path / d_T. All prompts are verbatim from upstream prompt/*.txt, assembled the way gpt.py:prompt_make does.
express_probeβ built-ins-only wiring probe: observe βbasic_agent__image_analyzecaption βllmCallproposes a waypoint JSON βtextParseβstep_pose. Exists to exercise the env end to end; its score is not meaningful. (Keeps the old always-truncate verdict band β it is not a paper baseline.)express_reβ the paper's Random Exploration baseline: stop check βsample_waypointrandom hop within 3 m + uniform yaw βstep_pose. Paper reference: RE C 36.95 / C* 62.75 / E_path 12.06 over the full 2,044 set.express_fbeβ the paper's Frontier-Based Exploration baseline: stop check βsample_frontier(TSDF fusion + frontier pick) βstep_pose. Paper reference: FBE C 38.60 / C* 63.03 / E_path 12.20.
Known deviation (both baselines): on budget truncation the judge image is one frame later than upstream's β the frame after the final hop rather than the frame at the final stop check. Fidelity to the initial view is exact the other way: reset now poses the agent with the record's start_rotation verbatim (upstream main.py:77+111), not an identity yaw.
6. Verification evidence
All rows on slot-b (/experiment:run, RTX 3090, ac-hmeqa env). The 30-episode rows are the first 30 val_unseen episodes β a small, easy-skewed subset, not comparable to the paper's 2,044-question rows.
| Check | Evidence | Result |
|---|---|---|
| Wire validation | POST /api/graphs/validate on express_probe | 33/33 edges wire-type clean; ensure_nodesets_for_graph auto-hosted the server-mode nodeset |
| First end-to-end run | run 20260729_130050 β probe, 1 val episode | completed: 4 teleport steps, 5th planner output unparseable β designed bad-action truncate β full verdict band (judge "Ξ΄, Ο" parsed, judge_ok=1) β C/C*/E_path/d_T harvested |
| Full-budget probe run | run 20260813_151848 β probe, 1 val episode, gpt-4o-mini active profile | completed: all 40 budget steps (clean budget truncation), judge_ok=1, Ο=2 β C*=40, Ξ΄=0 β C=0, d_T 7.01 m valid, path_len 262 m |
| RE baseline run | run 20260813_155735 β express_re, 1 val episode | completed: 40 random hops, no LLM in loop, verdict band OK β Ο=3 β C*=60, Ξ΄=0 β C=0, d_T 4.10 m valid, path_len 121.5 m. Same episode, random beat the probe's LLM waypoints on final distance β consistent with the paper's high RE floor |
| Faithful-loop smokes | runs 20260814_214556 / 20260814_214557 β reworked express_re / express_fbe, 1 val episode each | both completed with the stop module live: RE stopped early at step 6 (E_path 38.1), FBE at step 18 (E_path 21.4); stop_llm/stop_gate fired every step incl. the seed frame; answer_stop_llm (with-image branch) fired |
| RE 30-episode run | run 20260814_220732 β express_re, 30 val episodes, 6 workers | C 42.3 / C* 63.3 / E_path 25.5 / d_T 3.20 m; 13 early-stop vs 17 truncated episodes (both answer branches exercised). Paper full-set: C 36.95 / C* 62.75 / E_path 12.06 |
| FBE 30-episode run | run 20260814_220734 β express_fbe, 30 val episodes, 6 workers | C 47.7 / C* 67.3 / E_path 24.6 / d_T 6.60 m; 12 early-stop vs 18 truncated. Paper full-set: C 38.60 / C* 63.03 / E_path 12.20. C* tracks the paper closely on both baselines; C and E_path run high on this easy-skewed head subset |
The gpt-5-nano empty-response trap
A probe run under the shipped default active: gpt-5-nano profile (run 20260813_150526) died on step 1 looking like an env bug: caption, planner and judge all returned empty strings (reasoning tokens consumed the whole completion budget β the API succeeds with empty content), the parse produced an empty action, and step_pose took its designed bad-action stop. Diagnosis came from the per-episode log.jsonl: inner_log showed model: gpt-5-nano with 768 completion tokens and no text. With reasoning-family profiles either raise max_tokens well past 2000 or β simpler β run this benchmark under gpt-4o-mini, which is also its native judge.
Slot-pool artefact split (framework hole, observed here)
When a slot backend runs with AGENTCANVAS_EVAL_RUNS_DIR pointing at its own pool (outputs/eval_runs_b), the run-level artefacts (summary.json, spec.json, logs) land in the slot pool but the per-episode dirs (episodes/ep0000/log.jsonl + assets) land in the default outputs/eval_runs/ β the override doesn't reach the episode writer. Until fixed, look for episode logs under the default pool when debugging slot runs.
7. Not yet verified
- Full-set numbers. The largest run is 30 episodes (first-30 head of val_unseen); nothing is yet comparable to the paper's 2,044-question rows.
- Fine-EQA. The paper's own method (region-guided GOE + semantic value weighting, Prismatic VLM) is not ported; both detector-free baselines (RE, FBE) are. Upstream reference clone:
tmp/thirdparty/EXPRESS-Bench. - d_T on inf-geodesic episodes. The
d_t_valid=0path is implemented as an upstream mirror but no smoke episode has exercised it. - Canvas Play. Verified through the eval-batch path only; viewer nodes executed but never watched in the GUI.