SmartWay on AgentCanvas — paper analysis + smartway port

SmartWay (25.03; Shi et al., IROS 2025) is a zero-shot VLN-CE agent: a TRM waypoint predictor proposes candidate (angle, distance) moves from the panorama, RAM+ object tags enrich each candidate's description, and a backtracking return option lets the LLM undo a mistaken move. This page documents the decomposed port — the 7-node smartway reasoning nodeset plus the smartway_waypoint server sibling and generic FM wrappers (model_dinov2, model_ram ram_plus) driving env_habitat continuous navigation in smartway_ce.json. The validated 3-node monolith (smartway_mono) remains the equivalence ground truth.

1. Upstream method analysis (Shi et al. 2025)

Upstream SmartWay is a VLN-CE rollout (vlnce_baselines/common/base_il_trainer.py) around the same OneStagePromptManager class name MapGPT uses (vlnce_baselines/GPT/one_stage_prompt_manager.py) — but rewritten for continuous environments with waypoints, tags, and backtracking. Per step:

1. Predict waypoints. The panorama feeds a trained TRM waypoint predictor (waypoint_predictor/TRM_net.py, BinaryDistPredictor_TRM + DINOv2-small features per Policy_ViewSelection_VLNBERT.py:111) → K candidate (angle, distance) pairs, each with its view RGB.
2. Tag candidates. RAM+ (ram_plus, swin-L, image_size 384 — base_il_trainer.py:368–371) runs inference_ram on each candidate image (:440–442) → per-candidate object-tag strings.
3. Topology bookkeeping + synthetic return (make_action_prompt_backtrackv2, one_stage_prompt_manager.py:44–): the current position and each candidate are appended to nodes_list; the index in that list is the episode-global Place ID. last_distance is derived from the previous step's env action (:54–56); when the last step was not itself a backtrack and last_distance ≠ 0, a synthetic return candidate (angle π, distance = last_distance) is appended (:81–90).
4. Phrase each candidate (:108–112): "<direction> to Place N which is corresponding to Image N, and this image contains objects such as <tags>." — direction from six 60° buckets (get_action_concept_backtrack, :23–41). The trailing return candidate gets the verbatim phrase “Move back to last position in an opposite direction” (:110).
5. Letter options + stop gate (make_action_options_backtrack, :148–165): from t ≥ 2 (hardcoded) a bare stop is prepended; everything is letter-prefixed A/B/C….
6. Prompts (make_r2r_json_prompts, :202–245): a static 8-component task_description system prompt — including a dedicated Return Policy paragraph — plus a 4-field user prompt (Instruction / History / Previous Planning / Action options). gpt_infer_back_track (api.py:48–95) packs candidate images with "Image N:" labels before each image.
7. Parse (parse_json_action, :294–309): letter → index, else 0; at t ≥ 2 the index is decremented to absorb the prepended stop, so −1 means STOP. parse_json_planning (:284–291) extracts New Planning (fallback "No plans currently.") and appends it to the planning list.
8. Act (make_equiv_action, base_il_trainer.py:167–204): index → the candidate's (angle, distance), executed as rotate + repeated 0.25 m forward primitives.
9. History + backtrack latch (make_history, :168–184): appends "step t: <phrase>"; if the executed phrase string-equals the return phrase, self.backtrack = True (:183–184) — which suppresses the synthetic return next step.

1.1 Upstream invariants worth marking

• Place IDs are episode-global, but there is no map text. Unlike MapGPT, the prompt has no Trajectory/Map/Supplementary fields — nodes_list/graph/trajectory bookkeeping exists to mint stable Place IDs so history phrases written at step t still mean something at step t+k.
• The backtrack latch is an exact string match. make_history compares the executed phrase against “Move back to last position in an opposite direction” character-for-character (:183–184). Rewording the return phrase silently kills backtracking.
• The return option is self-suppressing. It is offered only when the previous step was not a backtrack and last_distance ≠ 0 — the agent cannot oscillate by chaining returns.
• Stop is gated at t ≥ 2 (hardcoded) and −1-encoded. Parse failure decodes as index 0, which after the shift means STOP late and first-candidate early.
• The system prompt is static. All episode state rides the 4-field user prompt; task_description (with the Return Policy paragraph) never changes across steps.

2. The AgentCanvas port

The port is a Phase-2 decomposition of the validated smartway_mono monolith (3 nodes), split along natural seams: plan_step → update_topology / build_action_options / assemble_prompt / build_images; decide_action → parse_response / resolve_action; update_history unchanged. Verbatim prompt constants and pure helpers live in smartway_mono/_prompts.py and are imported by both variants; test_equivalence.py checks the decomposition against the monolith. After the split, every graph_state key has exactly one writer.

2.1 Per-step flow

Drawn simplified in three places. ① The loop carry is minimal: iter_out collects step_hightolow.info as a pulse (persist=true) plus parse_response.is_stop on its stop port, and iter_in.step re-triggers observe_panorama — all agent memory crosses iterations through graph_state, the opposite of the MapGPT graph (which carries observations through iterOut). ② instruction enters once through iter_in's init_instruction slot (persist=true) and feeds assemble_prompt every step. ③ step_hightolow.terminated → update_history.step_done (an optional gate input) is omitted from the figure. Stop executes on both paths in the final iteration: parse_response.is_stop → iter_out.stop ends the loop (two-pivot model — no separate termination node), while resolve_action emits (0, 0) which step_hightolow treats as the canonical habitat STOP; after the loop, iter_out.final_stop triggers evaluate, harvesting the STOP step's metrics (the §4 rewire).

2.2 Node inventory

2.3 State & memory

The episode lifetime (vs MapGPT's run) means state clears between batch-eval episodes by container signal, with declared initial_values standing in at step 0. The two last_picked_* scratch keys are step-lifetime — per-step forensics, no cross-step reads.

2.4 Boundary contract

Reasoning is split from models and from the env. smartway (local) is pure reasoning; smartway_waypoint is a task-specific server-mode predictor in the ac-smartway conda env (Python 3.8.20, torch 2.1.1+cu121; SMARTWAY_PYTHON overrides the interpreter), and the foundation models ride generic wrappers: model_dinov2 (RGB features, ac-smartway) and model_ram (ram_plus tags, ac-ram). Env side: env_habitat via reset / observe_panorama (n_views=12) / step_hightolow / evaluate — the hightolow action space executes rotate + ⌊distance/0.25⌋ forward primitives, with (0,0) as the canonical STOP. Boundary-compliant per TODO #56's method/foundation-model split.

2.5 Prompt assets

All verbatim text lives in workspace/nodesets/method/smartway_mono/_prompts.py, shared by the monolith and this decomposition: the 8-component task_description (byte-diffed against upstream :205–232 — identical), the 4-field user-prompt template (:242–245), RETURN_PHRASE (:110), DEFAULT_PLANNING (:20), INIT_HISTORY (:234), and the pure helpers (direction buckets, options, parse). The graph wires assemble_prompt.task_description → llmCall.system_prompt — there is no duplicated prompt copy in the graph JSON.

3. Delta vs upstream — four buckets

A. Preserved verbatim

B. Forced by substrate (script → typed graph)

C. Forced by environment / cost

D. Intentional divergences

E. Unexplained / defects

None found (audit 2026-06-11). What was probed beyond the port's own claims: ① upstream's fuse_close_node=True signature default (one_stage_prompt_manager.py:44) is dead code — both rollout call sites pass False (base_il_trainer.py:446, :448) and the body never reads it, so the port's omission matches actual behaviour; ② the rollout calls make_history v1 (:509) with make_history_v2 commented out (:510) — the port correctly mirrors v1; ③ decomposition ↔ monolith I/O equivalence executed, not just cited: test_equivalence.py 10 passed in 0.17 s (byte-compares every output port and state write under a UUID-determinism fixture). Re-verify 2026-06-12: the reorg commit (119f82da) had left a stale from workspace.nodesets import smartway in the UUID fixture (it migrated the other four import sites); fixed, and the suite re-ran 10 passed in 0.18 s against the post-reorg tree.

4. Evaluation

Known gap (roadmap TODO #64 — confirmed on this graph): evaluate was a loop-body node triggered by step_hightolow.info; in the final iteration the loop terminated via parse_response.is_stop → termination before evaluate re-fired, so the harvested output_port__metrics holds the pre-STOP value. Habitat does compute success on the STOP step (visible in step.info.metrics): in run 20260610_091657, 5/20 episodes stopped within the 3 m radius but recorded success=0 → reported SR 0.000 vs true ≈ 0.25. Mechanism, not agent regression (nDTW 0.50 ≥ the 0.40 reference). Fix + re-verify before trusting any SR from the migrated graph. Update (2026-06-11 verify pass; committed 2026-06-11 as 7095f196): is_stop now feeds iter_out.stop and evaluate is triggered post-loop by iter_out.final_stop — the sound final-fire shape (trigger from the pivot, no upstream data-node dep). SR re-verification done 2026-06-13: run 20260613_115915 (rand100 20 ep) harvested SR 0.20 (4/20) — non-zero, every episode logging 3–15 decision steps, zero step_count=0 completions; the mechanism is fixed.

Operational gotcha (engine-level): at worker_count=20, the shared smartway_perception singleton returned HTTP 500 to 11/100 first-episode requests in run 20260516_101057 (cold-start race); GraphExecutor silently completed those episodes with step_count=0 and empty metrics (the silent-episode-completion engine hole). Check per-episode step_count before trusting aggregates from high-worker runs.

Current status: smartway_ce.json was promoted back to workspace/graphs/vln/verified/ on 2026-06-13 — the post-migration re-eval (run 20260613_115915: gpt-5-mini, rand100 20 ep, SR 0.20) confirmed the gym-rewired surface (observe_panorama / terminated / reset) and the two-pivot harvest behave correctly. A full rand100 (100-ep) run remains the headline-number target. The sibling monolith smartway_mono_ce.json was promoted to verified/ on 2026-07-05 via a no-LLM interface-equivalence audit: all 29 edges resolve against the current port declarations, its env-side wiring is handle-identical to the re-verified decomposed graph (so the gym-rewired surface and two-pivot harvest carry over), and the 10/10 mono↔decomp byte-parity suite still passes. The audit is pinned as smartway_mono/test_graph_interface.py.

5. Usage

# load — smartway (local) + the two server-mode model nodesets
POST /api/components/nodesets/smartway/load
POST /api/components/nodesets/smartway_waypoint/load?mode=server   # smartway conda env
POST /api/components/nodesets/smartway_perception/load?mode=server

# graph
workspace/graphs/vln/verified/smartway_ce.json

# batch eval — graph-only form: /experiment:run <profile> <graph_name> [key=value ...]
/experiment:run perf_smartway_ce smartway_ce          # rand100, 100 ep
/experiment:run smoke_smartway_ce smartway_ce episode_count=3

6. What this nodeset is NOT

• Not the monolith page. smartway_mono (3 nodes, smartway_mono_ce.json) is the validated Phase-1 port and the equivalence ground truth for this decomposition — same upstream, same prompts module.
• Not a PortBench v1 task. Deliberately excluded (Adelaide Qi Wu group authorship — author-relationship constraint); it lives as a side experiment and an AAS search target.
• Not a map-prompt method. Despite keeping nodes_list/graph/trajectory state, no map text ever reaches the LLM — the topology exists to mint stable Place IDs.
• Not env logic. Waypoint execution (rotate + forward primitives, STOP handling) lives in env_habitat.step_hightolow, not here.

7. Source files