NavGPT on AgentCanvas — paper analysis + navgpt / navgpt_mp3d_tools port

NavGPT (Zhou et al., AAAI 2024; repo GengzeZhou/NavGPT) is the original "LLM as navigator" agent: a GPT-4 ReAct loop picks viewpoint IDs on the MP3D graph via action_maker — the released config's sole tool, which already teleports to any viewpoint in the scan (back_tracer rides the non-default tool-chain path) — over text observations built from offline BLIP-2 captions and object detections. This page documents the navgpt_mp3d_tools port driving env_mp3d in navgpt_mp3d.json (the paper-faithful mainline), plus the in-house NavGPT-CE variant (navgpt.py + navgpt_ce.json, continuous env — a derived design, not a port).

1. Upstream method analysis (Zhou et al. 2024)

Upstream NavGPT is a langchain ReAct agent (nav_src/agent.py) on the MatterSim viewpoint graph. The released default in parser.py is use_single_action=True, use_tool_chain=False, use_relative_angle=True, load_instruction=True — a single LLM with everything in one prompt (VLN_GPT4_PROMPT for gpt-4, else VLN_GPT35_PROMPT), tool set [action_maker] only, and max_iterations=10 (parser.py:42, :17, :21). The modular multi-LLM chain (PLANNER_PROMPT / HISTORY_PROMPT / BACK_TRACE_PROMPT / orchestrator variants in the same file, the action_chain / history_chain / plan_chain + back_tracer tool) only fires under use_tool_chain — it is dead code at the default eval. Per step:

1. Offline perception (preprocessing, not in the loop): BLIP-2 ViT-G FlanT5-XL captions the egocentric views of every viewpoint; Faster R-CNN (Visual Genome vocabulary) detects objects within 3 m. The rollout reads these caches — no vision model runs at navigation time.
2. Observation text: per-direction scene descriptions + object lists + navigable viewpoints with relative orientation ("left 90", …) and distance.
3. ReAct prompt (VLN_GPT4_PROMPT): instruction + initial observation + the running scratchpad; the LLM answers Thought / Action / Action Input with tools action_maker or back_tracer, or stops with Final Answer: Finished! (agent.py:44).
4. Parse: Final Answer regex (agent.py:208) else Action/Action Input extraction; a missing Action Input is a format error fed back to the model.
5. Scratchpad (_construct_scratchpad, agent.py:117–130): appends each step's log; only the latest step keeps the full observation — earlier action_maker steps are demoted to one-line history summaries; the whole pad is tail-truncated to MAX_SCRATCHPAD_LENGTH = 7000 chars (:46, :136); init_observation is swapped to history[0] after step 0 (get_full_inputs).
6. Act: the viewpoint ID teleports the agent via MatterSim; back_tracer may target any previously visited viewpoint (non-adjacent allowed).

1.1 Upstream invariants worth marking

• No vision model runs at navigation time. Perception is an offline artifact; the loop is text-in, text-out. Any port that runs vision online is changing the method's cost profile, not just its plumbing.
• One prompt, several variants. planner_prompt.py ships the single-LLM VLN_GPT4_PROMPT/VLN_GPT35_PROMPT and the modular planner/history/orchestrator chain. The paper's headline GPT-4 number is the single-prompt configuration — "which prompt" is the first question any port must answer.
• The default tool set is action_maker only — and it already teleports anywhere. Under use_single_action the agent is built with tools=[self.action_maker] (agent.py:588); back_tracer exists only on the use_tool_chain path. And _make_action validates the ID against the whole-scan navigable_dict.keys() (agent.py:397), so action_maker can jump to any viewpoint in the scan, not just an adjacent one — back-tracing needs no separate tool.
• Anti-revisit is advisory upstream. The GPT-4 prompt merely "encourages" avoiding revisits by comparing IDs in previous Action Inputs — there is no hard constraint and no visited list rendered for the model.
• The scratchpad is lossy by design: only the newest step keeps a full observation, and the pad tail-truncates at 7000 chars — long episodes silently forget their oldest reasoning.
• Stopping is a string protocol: Final Answer: Finished! — there is no STOP action in the tool set.

2. The AgentCanvas port (navgpt_mp3d)

The mainline port reproduces the single-LLM VLN_GPT4_PROMPT configuration on env_mp3d, run on gpt-5-mini (upstream's GPT-4 is the reference; §3C). The ReAct prompt rides orchestrator_llm.config.template; the langchain agent loop becomes the graph loop (step_budget 10 = upstream max_iterations), with the scratchpad as an iterOut persist port and scratchpad_writer reproducing _construct_scratchpad's short-format demotion + 7000-char cap (_MAX_SCRATCHPAD_LENGTH, navgpt_mp3d_tools.py:1292). Perception is the paper's own offline artifacts, served by paper_objects_cache / paper_captions_cache nodes (1916-class object vocabulary, ~7.2 objects/viewpoint; BLIP-2 captions + summaries). The nodeset also ships online alternates (BLIP-2, InstructBLIP, COCO R-CNN, open-vocab detector) — none are instantiated in the reference graph.

2.1 Per-step flow

Drawn simplified: the init chain (reset → init_observe → init_format → init_observation, which builds the step-0 pad and history-0 line) and the duplicated step-side cache/observe/format chain are collapsed into single boxes; the caches gate the observe nodes via trigger wires so the formatter always sees cache text. Termination: step_waypoint.terminated → iter_out.stop (step budget 10 = upstream max_iterations); post-loop iter_out.final_stop → evaluate.trigger — the sound pivot-rooted shape. All agent memory crosses iterations through the two persist ports; this graph uses no graph_state container.

2.2 Node inventory (navgpt_mp3d_tools — 11 nodes)

2.3 State & memory

MP3D graph: scratchpad and init_observation are iterOut persist ports — the langchain agent's in-object state became pivot-carried wire values; nothing lives in a graph_state container. The cumulative visited-ID list is embedded in the scratchpad text itself (the "Visited Viewpoints" line), re-extracted and re-rendered each step precisely so it survives the 7000-char tail truncation (rationale recorded inline, navgpt_mp3d_tools.py:1671–1683). CE variant: parse_action appends {step, action, action_name, thought} to graph_state["action_history"]; format_history renders the last max_entries (default 15) with older steps elided.

2.4 Boundary contract

Local-mode nodesets (no server_python): the paper-cache and format/parse/scratchpad nodes are pure text plumbing; the optional online vision nodes lazy-load their models into the backend GPU. Env side: env_mp3d per ADR-server-003 (subprocess_per_worker, MatterSim batch 1) — step_waypoint keeps the navgpt-specific extras (success/error/turned_angle/new_viewpoint) that scratchpad_writer folds into the pad. The LLM is a plain llmCall; the ReAct prompt lives in graph config, not in a nodeset constant — prompt edits are graph edits.

2.5 Prompt assets

The system prompt is orchestrator_llm.config.template in navgpt_mp3d.json, derived from upstream VLN_GPT4_PROMPT (nav_src/prompt/planner_prompt.py). Verbatim-preserved: the embodied-agent framing, orientation conventions, the 3-m stop rule, the strict-viewpoint-ID warning, the full ReAct format block, and the closing scaffold (Thought: I should start navigation according to the instruction, {scratchpad}). Deliberately rewritten (recorded — §3D): the advisory anti-revisit sentence became a hard Visited Viewpoints contract, and back_tracer is reinstated with inline tool descriptions. CE variant prompts (5 llmCall templates) are in-house text with upstream echoes (its planner mirrors PLANNER_PROMPT's instruction-decomposition idea); the 3-node navgpt.py nodeset carries the CE action-format constant.

2.6 The NavGPT-CE variant (derived design, not a port)

navgpt_ce.json moves the idea to continuous VLN-CE — a setting upstream never shipped, so no fidelity verdict applies; it is audited only doc↔artifact. Structure: a 4-LLM pipeline per step — vlm_caption (panorama composite → scene text, temp 0.2), obs_summary_llm, orchestrator_llm (chooses FORWARD/LEFT/RIGHT/STOP; system prompt = the navgpt__action_output_format skill constant), history_update_llm, plus navgpt__parse_action (keyword regex, fallback FORWARD) into env_habitat.step_discrete; budget 150; termination step.terminated → iter_out.stop, post-loop final_stop → evaluate. It echoes upstream's modular chain (history LLM) rather than the GPT-4 single-prompt line. The previously-flagged dead planner_llm node was removed 2026-06-20 (§3E), trimming the graph to 25 nodes / 32 edges.

3. Delta vs upstream — five buckets (MP3D mainline)

Each row carries an equivalence icon (orthogonal to the bucket letter — the letter says why it differs, the mark says whether behaviour changed): 🟢 equivalent (identical or semantically identical behaviour) · 🟡 near-equivalent (same intent; a mechanism, library, or edge case differs — output usually matches) · 🟠 divergent (behaviour can differ — by design or, rarely, a defect). A genuine defect is a 🟠 row in bucket E with a severity — never ❌/🔴.

A. Preserved verbatim

B. Forced by substrate (script → typed graph)

C. Forced by environment / cost

D. Intentional divergences

E. Unexplained / defects

E bucket empty for the MP3D port. Probed cold from the upstream release this pass: the parser.py released config (single-action, action_maker-only, max_iterations=10) → mapped the live ReAct loop, not the dead use_tool_chain path; prompt re-diffed against VLN_GPT4_PROMPT — deltas localize entirely to the recorded anti-revisit / back_tracer region (§3D); modify_heading_angles formatting re-checked against _format_observation_compass (byte-identical on matched units); _construct_scratchpad demote + 7000-char cap + get_full_inputs init-swap matched (the swap is realized by the iterIn pivot wiring init_history_0 → iter_out.init_observation); step budget reconciled to max_iterations (15→10); stop-string set is a tolerant superset of Final Answer; paper caches are the released artifacts, not re-generated approximations. No equivalence unit test exists for either nodeset — eval parity is the only executable check, and none is currently on disk (§4); the next gpt-5-mini run is the evidence baseline.

4. Evaluation

Current status: both graphs sit in workspace/graphs/vln/unverified/ — rewired in the 2026-06-09 gym migration (bundle nodes → observe_* + step_waypoint/step_discrete; termination via iter_out.stop; evaluate post-loop via final_stop, the sound shape). The MP3D graph now drives gpt-5-mini (temp 1.0 / max_tokens 2000 / step_budget 10), so the next run is the evidence baseline — run /experiment:run navgpt-mp3d navgpt_mp3d episode_count=10. R2R val_unseen stratification applies to any future headline number (front-loaded easy scans — use the MapGPT72 cut for paper-comparable figures).

5. Usage

# load — method nodesets are local-mode; env_mp3d is server-mode (per-worker MatterSim)
POST /api/components/nodesets/navgpt_mp3d_tools/load
POST /api/components/nodesets/navgpt/load                     # CE variant nodes
POST /api/components/nodesets/env_mp3d/load?mode=server

# graphs
workspace/graphs/vln/unverified/navgpt_mp3d.json
workspace/graphs/vln/unverified/navgpt_ce.json

# batch eval — graph-only form: /experiment:run <profile> <graph_name> [key=value ...]
/experiment:run navgpt-mp3d navgpt_mp3d episode_count=10

6. What this nodeset is NOT

• Not NavGPT-2 / VLN-MME. This is the original AAAI-2024 NavGPT; later work by the same author is independent prior art, not covered here.
• Not a runtime-vision method (in the reference graph). The online BLIP-2/InstructBLIP/R-CNN nodes exist as alternates; the mainline reads the paper's offline caches, matching the paper's cost profile.
• Not a faithful port on the CE side. navgpt_ce.json is an in-house continuous-env derivation with its own prompts — no upstream counterpart, no fidelity verdict.
• Not env logic. Viewpoint-graph mechanics, teleport execution and metrics live in env_mp3d / env_habitat.

7. Source files