AO-Planner on AgentCanvas — paper analysis + aoplanner port

AO-Planner (24.07; Chen et al., AAAI 2025) is a zero-shot continuous-VLN (R2R-CE) agent built on Visual Affordances Prompting (VAP): a Grounded-SAM floor mask is grid-sampled into candidate pixels, a proposer VLM selects reachable waypoints, the pixels are unprojected to polar moves via depth + intrinsics, and a second PathAgent VLM picks the next location from four annotated direction views. This page documents the aoplanner port — a faithful two-VLM VAP pipeline driving env_habitat continuous navigation in aoplanner_ce.json, deliberately built to the paper's pure-foundation-model framing: the released code's supervised ETPNav/TRM waypoint predictor is dropped (the fidelity audit confirms it is dead-loaded at eval anyway). Both VLMs are gpt-5-mini.

1. Upstream method analysis (Chen et al. 2025)

AO-Planner is a VLN-CE rollout (vlnce_baselines/zero_shot_agent.py) whose action space is produced by two foundation-model calls per step, not a trained policy. The release run (run_r2r/zero_shot.bash → zero_shot_eval.yaml → LLM_base_task.yaml) renders 512×512 RGB and DEPTH (LLM_base_task.yaml:13–21) and caps the loop at 5 high-level steps (IL.max_traj_len 5, zero_shot.bash:17 — the bash override wins over the yaml's 15). Per step (zero_shot_agent.py:691, for stepk in range(max_len=5)):

1. Per-view affordance proposal (zero_shot_agent.py:708–740, four views front/left/backward/right at 0°/90°/180°/270°). For each view, grounded_sam_Gemini.py: GroundingDINO detects "ground" boxes (SwinT @ box/text 0.4), SAM masks them and unions all masks (:307–346); sample_points lays a 50 px grid over the union (up to 9×9 on 512) and prepends a foot point (W//2, H−5) as candidate 0 (:159–180, :348); >40 candidates (face-wall) or ==1 (no floor) abort the view (:354–362); otherwise the grid points are drawn as numbered dots (labels 1..N — the foot is never drawn) and a proposer VLM (Gemini-1.5-pro, temp 0) returns {Waypoints, Paths} (:267–304).
2. Unproject (utils.py:87–130): each chosen pixel → camera ray via depth + intrinsics → angle = atan2(x, d), planar distance = hypot(x, d). Depth is de-normalized ×10 (utils.py:26, habitat NORMALIZE_DEPTH=True, MAX_DEPTH=10); RGB and depth render at the same 512×512, so depth indexes 1:1 at RGB pixel coords (utils.py:115).
3. Ghost-graph bookkeeping (ZeroShotGraphMap, graph_utils.py:402–440): every candidate mints a fresh episode-monotonic ghost ID (ghost_cnt += 1, no merge); the current step's IDs form the PathAgent's option set — make_graph_baseline_prompts takes a ghost_pos argument but never reads it, so past ghosts are not offered as options. At eval this map is pure geometry — learned node/ghost embeddings are commented out, ghost_aug=0.
4. PathAgent decision (prompt_manager.py::make_graph_baseline_prompts:34–89): a static 10-piece task_description system prompt + a per-step user prompt (Instruction / History / Previous Planning / Options); the contributing (ghost-yielding) direction views are packed with "({dir}) Locations {ids} in Image {i}" captions (:58) at detail:"low" (GPT_api.py:69); GPT-4o (gpt-4o-2024-05-13, temp 0, max_tokens 600, json_object) returns {Thought, New Planning, Action}.
5. Parse + act: parse_num (:119–125) takes the first integer of Action; action.lower()=='stop' stops (:851); an ID beyond the episode's accumulated path list raises IndexError → stop (:838–844). The chosen ghost ID's Path (a multi-point pixel route) is executed by the env controller via multi_step_control — optionally with a try-out sliding obstacle-avoidance pass (environments_llm.py:447–526; the release sets ALLOW_SLIDING True, so use_tryout=False and habitat's native sliding does the job). History interleaves the actual past-step images at step>0 (prompt_manager.py:68–76). On the last step the selected path is walked and a STOP is stamped (zero_shot_agent.py:889–901). Eval metrics are hand-computed from the POSITION measure at episode end (:917–941) — see the §4 metric-ruler note.

1.1 Upstream invariants worth marking

• The "zero-shot" loop is genuinely zero-shot at eval. The supervised TRM (BinaryDistPredictor_TRM) and ETPNav policy (PolicyViewSelectionETP) are constructed and checkpoint-loaded (zero_shot_agent.py:183–199) but their forwards are never called in the rollout body (:636–986); align_with_waypoint_model (:579–634) returns empty candidate lists and is pure geometry. The action space comes entirely from the two VLM calls. (This is the audit's central finding — see §3D.)
• Depth is normalized; upstream de-normalizes it. Habitat's depth sensor is NORMALIZE_DEPTH=True, MAX_DEPTH=10, so the raw observation is [0,1]; utils.py:26 multiplies by 10 before unprojection. Skip this and every distance is 10× too small.
• Two VLMs, two roles. The proposer sees one view's candidate grid and chooses affordances; the PathAgent sees all four annotated views and chooses a numbered location. They never share a call.
• The PathAgent prompt is static + image-rich. The 10-piece task_description never changes; all episode state rides the user prompt, and history at step>0 is the actual past-step images, not text.
• Stop is the first-integer / 'stop' rule. parse_num takes the first digit run; a literal Stop (exact, case-insensitive) ends the episode. Format errors, no-candidate views, and an ID beyond the accumulated ghost set short-circuit to Stop (:818–844).
• The run-config chain is half the program. The paper run's effective values live in run_r2r/, not in code defaults: 512×512 RGB+DEPTH sensors, horizon IL.max_traj_len 5 (bash override), ALLOW_SLIDING True (→ use_tryout=False), VIDEO_OPTION ['disk'] — which is load-bearing, because the rollout reads its heading from the video-only top_down_map_vlnce['agent_angle'] measure, and MEASUREMENTS are stripped to ['POSITION','STEPS_TAKEN','COLLISIONS'] + hand-computed metrics (base_il_trainer.py:733, zero_shot_agent.py:917–941).
• Options are per-step; ghost IDs are per-episode. Ghost numbering never resets within an episode, but the PathAgent's option set is only the current step's ghosts (make_graph_baseline_prompts ignores its ghost_pos parameter). A history ID the agent re-names resolves through the episode-accumulated waypoint_path_coord — see §3D.

2. The AgentCanvas port

The port realises VAP as four replicated perception lanes (one per panorama view) that fan into a single SmartWay-style PathAgent decider. Verbatim prompts and pure parsers live in aoplanner/_prompts.py; geometry and overlay live in aoplanner/__init__.py. The supervised TRM/ETPNav stack is dropped (pure-FM framing); the ghost-graph is folded into a per-step aggregate manifest.

2.1 Per-step flow

Three simplifications in the drawing. ① The four perception lanes are identical — view0..3 → detect → ground_boxes → segment_box → sample → annotate → vlm1 → parse_proposal → project → make_bundle — and are collapsed into one row labelled "×4"; only their make_bundle outputs differ, fanning into aggregate.view_bundles (a LIST[ANY]) in lane order. ② propose_prep (the proposer system prompt, instruction-derived), observe_egocentric (intrinsics), and observe_camera_pose (agent pose for world-coord projection) are shared across all four lanes, not replicated. ③ Stop executes on both paths in the final iteration: parse_response.is_stop → iter_out.stop ends the loop (two-pivot model, no separate termination node), while resolve_action emits an empty hop list which step_path treats as the canonical habitat STOP (step(0)). Post-loop, iter_out.final_stop → emit_stop → final_step (step_hightolow) → evaluate: emit_stop feeds a constant (0,0) so a final step(0) registers an explicit STOP at the final pose (matching upstream's last-step move-and-stop; a no-op if the episode already stopped). evaluate harvests after that STOP.

2.2 Node inventory

16 tools; the four perception-lane tools are instantiated 4× in the graph (one per view). The proposer/PathAgent VLM calls are built-in llmCall nodes (5×), not aoplanner tools. pick_ground_box ships in the nodeset but is not instantiated by the reference graph (the lanes wire ground_boxes, which passes all boxes so SAM masks union over every ground box — the faithful path; pick_ground_box remains the single-box simplification variant, 0 graph refs).

2.3 State & memory

Three episode-lifetime keys. ghost_map stores only next_gid — a monotonic counter mirroring upstream ZeroShotGraphMap.ghost_cnt; no spatial dedup (E-1 fix 2026-06-19). The backtrack/tag topology that SmartWay carries is absent. Five nodes hold access_grants to graph_state: aggregate (writes ghost_map), assemble_prompt (reads history + planning), parse_response (writes planning), update_history (writes history + history_images), build_images (reads history_images).

2.4 Boundary contract

Reasoning is split from models and from the env. aoplanner (local, in-process) is pure reasoning + geometry; the only model-bearing dependency is the server-mode model_grounding_dino nodeset (GroundingDINO Swin-T OGC + SAM ViT-H) running in the ac-detany3d conda env (GROUNDING_DINO_PYTHON override). The two VLMs are built-in llmCall nodes. Env side: env_habitat via reset (env-override knobs RGB/DEPTH 512 + MAX_EPISODE_STEPS 5000) / observe_panorama (n_views=4) / observe_egocentric (intrinsics) / observe_camera_pose (world-frame position + quaternion) / step_path (in-loop; per-hop rotate + ⌊distance/0.25⌋ forward primitives, empty path = STOP) / step_hightolow (post-loop (0,0) STOP stamp) / evaluate. Boundary-compliant per TODO #56's method/foundation-model split.

2.5 Prompt assets

All verbatim text lives in workspace/nodesets/method/aoplanner/_prompts.py: the proposer system prompt (5 pieces + instr_des, build_proposer_system), the PathAgent task_description (10 pieces, build_task_description), the per-step user prompt (assemble_pathagent_prompt), the image caption (image_label), the parsers (parse_proposal, parse_pathagent), and the seed strings (DEFAULT_PLANNING, INIT_HISTORY). All five were byte-diffed against the pinned upstream and are identical (§3A). The graph carries no duplicated prompt copy; the vlm1 template is a minimal JSON-nudge (port glue — llmCall requires non-empty user text).

3. Delta vs upstream — five buckets

Each row carries an equivalence icon (orthogonal to the bucket letter — the letter says why it differs, the mark says whether behaviour changed): 🟢 equivalent (identical or semantically identical behaviour) · 🟡 near-equivalent (same intent; a mechanism, library, or edge case differs — output usually matches) · 🟠 divergent (behaviour can differ — by design or, rarely, a defect). Never ❌/🔴 — a divergence is a design fact; a genuine defect is a 🟠 row in bucket E with a severity. Audited 2026-06-17, inventory-first, against chen-judge/AO-Planner @ 719f42a1.

A. Preserved verbatim

Behaviour is equivalent (🟢) or near-equivalent (🟡). Equivalent items are A regardless of mechanism difference — a substrate-relocated item is A, not B.

B. Forced by substrate (script → typed graph)

Mechanism must differ due to AgentCanvas's typed-graph constraints; behaviour may or may not be equivalent. (The former Options-text-position row was fixed 2026-07-04 — the line now rides the first option image's label, reproducing upstream's content order — and moved to §3A.)

C. Forced by environment / cost

(The model substitutions themselves are choices, not constraints — gpt-4o is API-accessible — so they moved to §3D with their recorded rationale; what stays here is what the chosen model forces.)

D. Intentional divergences

E. Unexplained / defects — none open

Probed (2026-07-04, iteration 3, source-first): the release run-config chain (run_r2r/zero_shot.bash → zero_shot_eval.yaml → LLM_base_task.yaml → base_il_trainer.eval()/_set_config effective values) · zero_shot_agent.py full rollout including the hand-computed metric block (:917–941) · grounded_sam_Gemini.py full · llm/utils.py full · prompt_manager.py full · GPT_api.py/Gemini.py (packing, detail, sampling) · environments_llm.py full (step dispatch, only_stop/act-0 semantics) · graph_utils.py ZeroShotGraphMap · measures.py POSITION/TopDownMap agent_angle · collect_val_traj episode-set construction · fork-diff vs parent ETPNav (README lineage; deltas = the new llm/+zero_shot_agent+environments_llm+run_r2r/ files and comment-level edits to inherited files — nothing live hides in the inherited tree). Iteration 3 surfaced 8 findings (512×512 render, horizon 5, empty-view leak, out-of-range STOP, count-mismatch branch, proposer label scheme, route-drawing style, MAX_EPISODE_STEPS) — all fixed 2026-07-04 and re-filed to §3A rows above; earlier passes' findings (2026-06-17: 7; 2026-06-19: 3) are likewise closed into §3A/§3D. Zero open rows; history in the changelog.

4. Evaluation

Metric rulers, pinned on both sides. Upstream hand-computes its metrics from the POSITION measure at episode end (zero_shot_agent.py:917–941): success = final geodesic ≤ 3 m after the episode's STOP (every exit path stamps one, incl. the last-step move-and-stop), SPL = success·d₀/max(d₀, path_length) with d₀ = the start geodesic (distances[0]), nDTW = exp(−dtw/(len(gt)·3)) against {split}_gt.json.gz locations, oracle = any low-level-step distance ≤ 3. The port reads habitat's official VLN-CE measures (evaluate → SUCCESS/SPL/NDTW/ORACLE_SUCCESS/PATH_LENGTH), whose formulas are the same expressions — success is stop-gated (the port's emit_stop chain stamps the STOP), SPL's shortest-path term is the same start geodesic, and measures update per low-level primitive (same trajectory granularity, since step_path drives real env.step calls). The two rulers are formula-equivalent; the one dataset-level caveat: our env serves R2R_VLNCE v1-3 (VLN-CE repo default), upstream's yaml points at v1-2 — same task, slightly different episode/gt revisions, so cross-checking against the paper's exact number additionally rides that revision.

The depth-normalization fix (between the two runs above). The first real eval scored SR 0 with every hop ≤ 1 m. Root cause: env_habitat emits normalized [0,1] depth (habitat NORMALIZE_DEPTH=True, MAX_DEPTH=10), and the port decoded it as metres — exactly the de-normalization upstream does at utils.py:26. Angles were correct (scale cancels in atan2), only distances were 10× short. Adding depth_scale=10 (+ a depth resize to the RGB/intrinsics resolution — moot since the 2026-07-04 native-512 alignment) lifted SR 0→0.10, oracle 0→0.30, path 4→21 m. Both changes are upstream-parity restorations (§3A), not new behaviour.

Sampling variance dominates at this N. The §3E fixes are in (run 20260617_122148), but their SR effect is below the noise floor: on the shared first-10 val_unseen episodes, headline SR held at 1/10 while SPL rose 0.036→0.052 and nDTW 0.26→0.40 — and the per-episode outcomes flip run-to-run (ep2 ✗→✓ at d2g 9.05→1.53; ep8 ✓→✗ at 2.21→11.67; ep1 d2g 1.72→12.30 on the same seed). Cause: gpt-5-mini is forced to temperature=1.0 (the gpt-5 family returns empty otherwise — no deterministic mode), so at 10–20 episodes the run-to-run variance exceeds any fix/tuning delta. A trustworthy headline needs the 216-ep MapGPT72 slice or multi-seed averaging; treat SR ≈ 0.10 as a noisy current estimate.

Known gaps. (1) Resolved 2026-07-04: the graph is promoted to vln/verified/ on the 100-ep random-slice headline (SR 0.35, above). (2) Paper-comparability: any claim about the gap (in either direction) vs the paper's number rides the §3D model substitution and the v1-2→v1-3 dataset revision — attribution to model tier is an unverified hypothesis until a Gemini-1.5-pro + gpt-4o control run. (3) The §3D fallback-vs-replay residue (past-ghost IDs) fires only when the PathAgent names a stale ID — rare but real. (4) Single-seed at forced temp 1.0: multi-seed averaging would tighten the headline.

5. Usage

# load — aoplanner (local, in-process) + the Grounded-SAM server nodeset
POST /api/components/nodesets/aoplanner/load
POST /api/components/nodesets/model_grounding_dino/load?mode=server   # detany3d conda env

# run (graph-only form; JobScheduler admits by profile)
# step_budget=5 = the paper horizon (IL.max_traj_len 5); per_step_budget_sec
# covers the ~75 s/step of the 512-px faithful config (4x Grounded-SAM + 5 VLM calls)
/experiment:run aoplanner-ce aoplanner_ce split=val_unseen episode_count=10 worker_count=10 step_budget=5 per_step_budget_sec=150

Graph: workspace/graphs/vln/verified/aoplanner_ce.json. Profile aoplanner-ce (vram 8000, non-exclusive) in .claude/experiment/profiles.yaml. The four perception lanes serialise on the single-GPU Grounded-SAM server, so high worker counts queue there.

6. What this nodeset is NOT

• Not the released AO-Planner pipeline. The supervised TRM/ETPNav stack is dropped (pure-FM framing); only the two VLM calls produce the action space. Numbers compare only against the paper's zero-shot line, and even that with the §4 caveats (model substitution, split, dataset revision) — never against the distilled SR-47 variant.
• Not a full ghost-graph navigator. Ghost IDs are episode-monotonic with no spatial dedup — exactly upstream's ZeroShotGraphMap minting — and the option set is current-step-only in both (upstream never offers past ghosts either, §3A). What the port does NOT keep is upstream's episode-accumulated per-ghost path store: a stale ID falls back instead of replaying the old route (§3D, user-deferred). The multi-point path executor is restored (D-2, env_habitat__step_path).
• Not SmartWay. It clones SmartWay's decider structure (assemble / build_images / parse / resolve / update_history) but drops backtracking, RAM+ tags, and the TRM predictor; the perception half is AO-Planner's VAP.
• Not Gemini/GPT-4o-locked. Both VLMs are profile-driven llmCall nodes, pinned to gpt-5-mini in this graph.

7. Source files