MapGPT on AgentCanvas — paper analysis + mapgpt port

MapGPT (24.01; Chen et al., ACL 2024) is a zero-shot VLN agent: it embeds a linguistic-form topological map of explored/observed places directly in the LLM prompt and carries adaptive multi-step planning text across steps. Section 1 dissects how the upstream OneStagePromptManager pipeline actually works; Section 2 documents the AgentCanvas port — the 8-node mapgpt nodeset driving env_mp3d discrete panoramic navigation; Section 3 files every difference into the four delta buckets.

1. Upstream method analysis (Chen et al. 2024)

Upstream MapGPT is a rollout script (vln/gpt_agent.py) around one stateful prompt manager (GPT/one_stage_prompt_manager.py). Per step:

1. Observe. The simulator returns the current viewpoint, heading, and the navigable candidate viewpoints, each with a per-view panorama image keyed by pointId.
2. Grow the map. The manager folds the current viewpoint and candidates into four parallel structures — nodes_list (every place ever seen; index = Place ID), graph (adjacency, recorded on first visit only), trajectory (visited sequence), node_imgs (one cached image per place).
3. Serialize the map into language (make_map_prompt, lines 124–165): a Trajectory line of Place IDs, a Map block of “Place i is connected with Places j, k” lines, and Supplementary Info — places seen but never visited (the backtracking menu). Returns "Nothing yet." when empty (line 163).
4. Build action options (make_action_prompt / make_action_options): each candidate becomes a phrase "<direction> to Place i which is corresponding to Image i", where the direction word (“turn left”, “go forward”, “turn around”, “go up”…) is computed from the heading delta by get_action_concept (lines 16–39). Options are letter-prefixed A./B./C.… (make_action_options, lines 95–112); from t ≥ stop_after a bare stop option is prepended (lines 103–105). The canonical action list always carries stop at index 0 (make_history, line 117), so while stop is not yet offered parse_json_action compensates by adding +1 to the parsed index (lines 349–351) — the “index-shift trick”.
5. Assemble the 7-field user prompt (make_r2r_json_prompts, lines 218–236): Instruction / History / Trajectory / Map / Supplementary Info / Previous Planning / Action options, plus every cached place image packed as "Image i:"-labelled inputs aligned to Place IDs (GPT/api.py:22–44).
6. Call GPT in JSON mode. The system prompt demands a 3-field JSON response: {"Thought", "New Planning", "Action": "<letter>"}. parse_json_action (lines 336–352) reads the letter; New Planning is stored and echoed next step as Previous Planning — this is the “adaptive planning” carry.
7. Act. Letter → candidate viewpoint; the executed action phrase is appended to History as "step t: <phrase>" (make_history, lines 114–122). A hard budget aborts the episode with STOP when the packed image list exceeds 20 (vln/gpt_agent.py:133–136).

The row layout deliberately mirrors the port diagram in §2.1 — each upstream function sits where its corresponding canvas node sits, so the two figures can be compared box-by-box.

1.1 Upstream invariants worth marking

• Place i ↔ Image i alignment is load-bearing. The map text, the options text, and the image labels all reference the same index into nodes_list. Any port that re-orders or re-keys places silently breaks the LLM's grounding.
• Stop is a gated option, not a free action. Before the threshold the LLM cannot stop; after it, stop is option A and everything shifts. The parse fallback (defaulting to index 0) therefore means “stop” late in the episode and “first candidate” early — intended behaviour, not a bug.
• Adjacency is first-visit-frozen. graph[vp] is written once, on the first visit to vp — revisits do not update it.
• History records action phrases, not viewpoints. What enters History is the chosen option's phrase (“turn left to Place 3 …”), the same string the LLM saw — keeping its self-model consistent.
• The image budget is a hard abort. >20 cached images ⇒ forced STOP. Upstream's stated reason is an API limit — “GPT-4o currently does not support queries with more than 20 images” — though it also bounds prompt cost and episode length.

2. The AgentCanvas port

workspace/nodesets/method/mapgpt.py splits the monolithic prompt manager into a per-step pipeline of single-responsibility nodes. The split point that matters: direction phrases need both the current heading and each candidate's heading, which only coexist at the env boundary — so observe computes them immediately and freezes them into candidates_json; downstream nodes never need heading access. (An earlier split that deferred direction computation to a later stage lost the current heading and was abandoned — see the module docstring.)

2.1 Per-step flow

The loop is a standard iterIn/iterOut scope, drawn simplified: observations enter observe through iterIn's dual slots (init_* from seed_nav/seed_pano at step 0, iterout_* thereafter), and step.info triggers the loop_nav/loop_pano re-observations that iterOut collects for the next iteration. Stop: parse_action emits viewpoint_id="STOP", which step_waypoint answers with terminated=true, wired into iter_out.stop (since 2026-06-11; env_mp3d treats STOP as episode-done without moving). image_budget.done is the second writer to iter_out.stop — either signal ends the loop. evaluate now sits in the after-loop band, triggered once by iter_out.final_stop at termination.

2.2 Node inventory

2.3 State & memory

2.4 Boundary contract

Reasoning-only nodeset — no simulator calls, no model weights. Env side: env_mp3d via reset / observe_navigable / observe_panorama / step_waypoint / evaluate (waypoint action space, navigable + panorama obs spaces per the env contract). FM side: the built-in multimodal llmCall, model selected by LLM profile — no vendor lock-in in the nodeset. This page is boundary-compliant per TODO #56's method/foundation-model split.

2.5 Prompt assets

All prompt text lives as constants in mapgpt.py: the system prompt (verbatim from one_stage_prompt_manager.py:218–236), init-history / default-planning / no-supplementary strings (lines 189, 14, 163). The 7-field user prompt is assembled at runtime by render_prompt with upstream's exact field order and labels. The reference graph additionally duplicates the system prompt byte-identically into planner_llm.config.system_prompt (verified 2026-06-11) — when editing one copy, update the other.

3. Delta vs upstream — four buckets

A. Preserved verbatim

B. Forced by substrate (script → typed graph)

C. Forced by environment / cost

D. Intentional divergences

E. Unexplained / defects

Inventory-first audit, 2026-06-11 — three findings, none with recorded intent (hence E, not D):

Probed and cleared (adjudicated non-issues): ① previous_angle is the agent's heading at prompt time (set from fresh obs at gpt_agent.py:93/:167) — the port's cand_heading − cur_heading is equivalent despite the misleading upstream name; ② upstream stores plannings as a list but every prompt reads only [-1] (:200/:202, :285) — the port's lastWrite string is behaviour-equivalent; ③ make_history is live on the json path (:174); ④ packed image_list = all cached place images with None skipped — matches render_prompt; ⑤ upstream loop budget max_action_len=15 (parser.py:16) maps to the eval-side step budget, not a nodeset constant. No mapgpt equivalence test exists to run (unlike SmartWay) — eval-parity is the only executable check.

4. Evaluation

Comparability caveat (from §3E): upstream sends every image at detail:"low"; our llmCall sends no detail parameter, so the port's model sees higher-fidelity images than the paper's. Treat the +9 pp closure margin as partially confounded until the detail divergence is resolved.

Known gap — closed structurally (2026-06-11): the loop-body evaluate pattern that under-counted SR after the terminal STOP step (roadmap TODO #64) was eliminated by the two-sided-iterOut refactor: evaluate is now after-loop, fed exactly once at termination via final_stop, so the harvested output_port__metrics reflects the terminal step by construction. The 2026-06-13 re-eval confirms it: run 20260613_120854 harvested SR 0.55 (11/20) with every episode logging real decision steps and zero step_count=0 completions.

Current status: mapgpt_mp3d.json was promoted to workspace/graphs/vln/verified/ on 2026-06-13 after the post-migration re-eval (run 20260613_120854: gpt-5-mini, MapGPT72 20 ep, SR 0.55 reproducing the closure number) confirmed the gym-rewired surface (observe_navigable / observe_panorama / step_waypoint) and the two-pivot harvest behave correctly. A full 216-ep MapGPT72 run remains the headline-number target.

5. Usage

# load (canvas: NodeSet Manager page, or REST)
POST /api/components/nodesets/mapgpt/load          # local mode; env_mp3d loads in server mode

# graph
workspace/graphs/vln/verified/mapgpt_mp3d.json     # open in canvas, or submit via eval

# batch eval — graph-only form: /experiment:run <profile> <graph_name> [key=value ...]
# (the legacy "profile -- command" form was removed 2026-05-07)
/experiment:run perf_mapgpt_mp3d mapgpt_mp3d split=MapGPT72 episode_count=216 worker_count=4

6. What this nodeset is NOT

• Not the metric-grid MapGPT (roadmap M2): that variant builds an occupancy map from depth. This nodeset is the paper's linguistic-topo-map method, used for PortBench v1.
• Not env logic: no MatterSim calls, no episode control — all simulator interaction goes through env_mp3d wires.
• Not model-locked: the upstream gpt-4o pin became an LLM-profile choice; the prompt contract is the only model-facing assumption.

7. Source files