Roadmap

First monolith smoke (3 ep) exposed a metric/termination gap in tooleqa__step + workspace/graphs/eqa/unverified/tooleqa_hmeqa.json: episodes where the 3B agent never calls final_answer run to the env step-limit and emit no metrics port, so they're silently excluded from the SR aggregate (harness reported 1.0 = 1/1 metric-producing episode, true result 1/3). Three fixes: (1) the max-iter fallback keys off env step_index, which only advances on go_next — switch it to the ReAct step count so non-terminating episodes still emit a (failed) answer + metrics; (2) metrics.num_steps reports 0 — populate with the ReAct step count; (3) count env-done-without-answer episodes as success=0 rather than dropping them — partially covered since 2026-06-11: the batch runner now flips an eval episode with no graphOut snapshot to status=error (no more silent exclusion), but emitting an explicit failed verdict is still the right fix. Also investigate why the 3B keeps exploring instead of answering (prompt/format adherence vs upstream 7B).

Nodes + graphs exist (workspace/nodesets/method/navgpt.py + navgpt_mp3d_tools.py, graph JSONs); remaining: agent manifest, skills module, consolidate two graph variants, clean organization (merged with former #27).

Verify the functionality of our future state container system — identify a published agent architecture whose reasoning loop requires shared persistent state across steps. Implement as graph preset using state containers to validate the dual-wire design before broader rollout.

Low priority. LoopRunner is 115 lines with clean separation; GraphExecutor well-structured. Revisit only if bugs surface in IterIn/IterOut, pause/resume/stop lifecycle.

in-memory checkpoint wiring at IterOut (done), checkpoint/restore API (done), migrate HistoryTrackerNode/LLMCallNode/NavGPTReasonNode to graph_state (done), NavGPT-CE graph JSON with graph_state (done), state config panel in the frontend (done — StatePanel.tsx: add/remove named states, reducer type). Remaining: decompose HistoryTracker into composite graph with visible state, implement container checkpoint persistence to disk.

Current frontend logic is fully Claude-generated; audit canvas, API client, store, and node types against actual backend endpoints and models to fix any drift or assumptions.

Audit and reorganize the top-level directory layout for clarity and long-term scalability: review workspace/ subdirectory conventions, evaluate whether agentcanvas/backend/app/ structure scales, align folder naming with documented concepts. Define a canonical layout doc.

Custom method nodes are proliferating (NavGPT/MapGPT/DiscussNav each ship their own parse/format/reason trio), making the canvas look like ordinary visualization over one-off code. Framework abstractions are solid; the domain layer is thin. Candidates: parameterized reasoning turn (prompt template + LLM + structured parser), history formatter with pluggable strategies (full / last-k / summarized), action-space adapter (discrete panoramic ↔ continuous waypoints), observation encoder (pose + RGB + depth → text/image bundle). Do the extraction pass against NavGPT + MapGPT + DiscussNav + VLN-Zero concretely.

Follow-up to #10. Every node in habitat.py, sam.py, policy_cma.py, others.py currently only sets NodeUIConfig(color="..."). Audit which nodes should expose inline editable config_fields and display_fields. Per node: color-only is fine, or upgrade to block/strip/viewer layout. One-line classification per node, then edits for ones that need the richer form.

Different env nodesets expose radically different action ports today (MP3D: dynamic-cardinality discrete viewpoints via viewpoint_id TEXT + cand_vpids_json; Habitat-continuous: 2-vector or Discrete(4) ACTION int; VLN-CE: discrete Discrete(4); future AI2-THOR: parameterized actions with object IDs), which forces every agent graph to be env-specific. Proposal (deferred): every env's reset/step emits a single action_manifest (TEXT, JSON-serialized) with a first-field type discriminator describing the currently valid action set. For now: keep env-native shapes; revisit once a third env nodeset lands and we can see the real minimal common schema.

Today errors are silent on two paths: (a) server-mode nodes' _self_log("error", ...) is not carried back over the proxy HTTP response; (b) many built-in tools swallow soft failures into empty default outputs without exposing an error output port.

Scope TBD (likely covers workspace/{graphs,graph_nodes,nodesets,policies} discovery, outputs/eval_runs/ vs data/outputs/{eval,eval_mock,eval_native}/ vs legacy outputs/runs/ JSONL persistence, overlapping ownership with TODO #40 — clarify what "file management" covers before scheduling).

Under worker_count > 1 with a batched=True node hosted in a parallelism="shared" server-mode subprocess, _BatchQueue._delayed_flush raises RuntimeError: got Future attached to a different loop (observed 2026-05-07 on policy_vlnce__predict at K=10; 88/100 episodes returned metrics={}). Workaround in tree: policy_adapter_vlnce__predict.batched = False (node renamed from policy_vlnce__predict in the 2026-07-04 split). Real fix: make _BatchQueue loop-agnostic or per-worker.

Current architecture erases composite at compile time via flatten_graph — leaky abstraction. Cleaner: composite stays as a real runtime entity, executor walks composite boundaries natively. Trade-off: flatten = simple executor + complex compile-time transform with leaky patches; composite-native = composite-aware executor + simple boundary resolution. Trigger: third leak shows up, OR multi-scope authoring becomes the dominant pattern. Until then, the conditional-strip middle option keeps things working.

13 install_*.sh in scripts/install/ plus 11 conda envs on disk. Naming inconsistent (agentcanvas-X vs bare vlnce/hmeqa/vlaws*), env→nodeset mapping implicit, several install scripts duplicate boilerplate. Audit goals: document canonical env↔nodeset table; decide which envs can merge; extract shared install boilerplate; clarify install_all variants or collapse them; align naming.

Each v1 agent form should have its own tutorial page under docs/pages/developer-guide/tutorials/ showing a minimal worked example: DAG / cyclic loop / ReAct hidden / ReAct router / Plan-and-Execute / bounded multi-agent / embodied sense-act / FM pipeline. Canonical list lives in major-versions.html §1 "Agent forms v1 covers".

Opening a graph synchronously activates its nodesets. The frontend now shows a loading overlay + Cancel, but Cancel only aborts the client fetch — it does not stop the backend load. True hard-cancel is only possible at the process boundary = server mode: a live GPU/CUDA object's ownership is bound to the process that created it and cannot be handed back to the parent, so "load in a subprocess then return the object" fails for GPU models (works only for pure-CPU picklable objects, which rarely load slowly enough to need interrupting). Plan: (a) offload the synchronous BaseServer.start() to a thread, fixing the framework hole where server-mode startup blocks the entire event loop for up to startup_timeout (1800s); (b) run the ensure endpoint as a token-keyed asyncio.Task + add POST /nodesets/ensure/cancel, where server-mode cancel kills the subprocess; (c) auto-promote heavy local nodesets to server mode so they gain hard-cancel; lightweight local stays on client-side fetch abort (soft cancel).

First full rewrite of docs/pages/developer-guide/design-docs/graph/graph-executor.html landed 2026-06-08 (Part I concept SVGs + flowchart; Part II verbatim run()/_fire_node/helpers with ◀── annotations). Committed as a checkpoint — refine later: tighten prose, check dark-mode contrast and flowchart arrow routing in-browser, consider trimming the full-verbatim listings if they prove too long to maintain. 2026-06-11 (evening): the listing problem is structurally resolved — Part I split into graph-view/engine-view, §4.5 redrawn around the ready-queue, §4.6 now a turn-by-turn table; Part II slimmed to run() only (_fire_node/helpers read in source) and the listing is synced from source by docs/_lib/_sync_run_listing.py (comments are the annotations; Pygments two-stage highlighting per .claude/standard/code-highlighting.md). Remaining: in-browser pass — dark-mode contrast, flowchart arrow routing, the new GitHub-palette listing.

After the env-nodeset gym-like refactor (2026-06-09), the per-step evaluate node (triggered by step_*.info) does not fire after the agent's final STOP step — the loop terminates (e.g. parse_response.is_stop → termination) before evaluate re-fires — so the harvested output_port__metrics graphOut holds the pre-STOP value (success=0). Confirmed on smartway_ce (20 ep, gpt-5-mini, step_budget=15, run 20260610_091657): habitat computes success=1.0 on the STOP step (visible in step.info.metrics) but 5/20 episodes that stopped within the 3 m radius (ep 9/10/11/14/17, dist 0.43–2.61 m) recorded success=0 → reported SR 0.000 instead of ≈0.25 (baseline 0.28). Mechanism, not agent regression (nDTW 0.50 ≥ baseline 0.40; oracle 0.40). Structural fix landed 2026-06-11 (two-sided iterOut refactor): the termination node was removed, evaluate moved to the after-loop band fed once at termination via the iterOut's final_stop handle, and the validator now rejects loop-body-fed metrics/success graphOuts — the engine-level regression test asserts the after-loop consumer receives the terminal iteration's value. Remaining work = re-verification: smartway_ce (re-verified — promoted to vln/verified/), straightforward (re-verified 2026-06-28: bit-identical to native CMA, 50-ep SR 0.38 / SPL 0.348 — promoted to vln/verified/); 2026-07-05 wave: smartway_mono_ce (interface-equivalence audit — env wiring handle-identical to re-verified smartway_ce), navgpt_mp3d (dead legacy state edges removed; clean 1-ep smoke 20260705_194214, full metric harvest via final_stop), octo_simpler (clean 1-ep smoke 20260705_193758) — all promoted to verified/; still open: navgpt_ce (llmCall action/summary port drift) and opennav_habitat (llmCall template drift) — the rest of the wave has landed (threestepnav_ce and octo_simpler now sit in verified/; discussnav_mp3d / spatialnav_mp3d are tracked by #73 / M13, not this mechanism bug). See env template.

Surfaced 2026-06-11 while smoke-testing the gym-migrated vla_policy_libero.json. The graph's ✅-verified datapoint (2026-05-02, 5/5) came from scripts/smoke/smoke_vla_libero.py against the external vlaworkspace; the canvas-graph path through the policy_adapter_vla server has been broken since the 2026-05-04 adapter split. Debt chain, outermost first — first three fixed 2026-06-11: (1) graph checkpoint_path pointed at a deleted vlaworkspace slurm dir → repointed to vendored data/vla_policy/checkpoints/pi0_libero_pytorch; (2) LiberoRobot lost its RobotAdaptor base in the adapter split, so ensure_robot's _find_subclass never matched → re-based; (3) adapters/models/pi0_model.py kept a one-level-short relative import (..models.openpi → resolves to adapters.models.openpi) → ...models.openpi; (3b) ac-vla-policy env had pip nvidia-nvjitlink-cu12 12.5 vs 12.9-era CUDA libs → upgraded to ~=12.9. Remaining blocker: Pi0Policy.__init__ with DEFAULT_KWARGS use_pretrained_weight=True tries a JAX→PyTorch conversion via python -m vlaworkspace.model.convert_jax_model_to_pytorch (module not vendored; cache_dir 'data/models' also doesn't match the vendored layout data/vla_policy/checkpoints/pi0_base_pytorch) — even though the finetuned safetensors is already local. Fix = make the pretrained-base path resolve to the vendored pi0_base_pytorch (or skip base-init when a full finetune checkpoint is given), then smoke vla_policy_libero 1 ep. Companion engine hole (cousin of the silent-episode-completion memory): a server proxy node whose response is {"x": null} gets logged as outputs={} with error=null and the episode "completes" at step_count=0 — three of today's four failures were silent for this reason.

A follow-up cleanup that surfaced from the #67 transport work. env_habitat's panorama path (observe_panorama, env_habitat.py:536-537) emits two depth products per view: depth_base64 (8-bit min-max-normalised PNG via encode_depth_base64, viewer-only, lossy / relative) and depth_raw_base64 (16-bit-millimetre PNG via encode_depth_raw_base64 env_habitat.py:798, metric-faithful, capped at 65.535 m to fit uint16). This base64-PNG encoding is an application-layer contract consumed by downstream method nodesets (opennav_perception._decode_depth_raw, aoplanner._decode_depth_m, smartway_waypoint) — distinct from, and predating, the Move-1 msgpack transport (#67), which can now carry a raw ndarray across the env boundary as a blob ExtType with no base64 at all. Not a bug — kept intentionally because (1) the consumers' wire contract is a base64 string; (2) 16-bit PNG compresses the depth map (a raw float32 480×640 panorama set is large); (3) the "16-bit / mm / cap 65.535 m" metric semantic is carried by the I;16 PNG format choice; (4) it survives the JSON legacy path too. Cleanup question: once the JSON legacy path is retired (transport msgpack-only) and the consumers can be migrated together, depth_raw_base64 could collapse to a raw depth ndarray + a units flag (is_mm/metres), letting msgpack own compression/encoding and removing the bespoke encode_depth_raw_base64 / _decode_depth_raw pair. Scope: weigh the bandwidth win of PNG compression against the simplification before scheduling; relates to #67 (transport contract) and the viewer-vs-metric depth split.

Scale batch eval from a single machine to coordinated multi-machine execution (an AgentCanvas open-source prerequisite, from the find-job dependency graph in the personal todo plan). Prerequisite: first map out the current batch-eval implementation (single-machine worker pool / JobScheduler status quo, the 8765–8769 port pool, admission control) before designing cross-machine scheduling.

The three support-status pages (vln-/eqa-/vla-support-status.html) drifted while being converted onto the VLN two-part template (2026-06-30). Divergences to reconcile: VLN's §1 matrix uses CSS-styled ours-* badge spans plus an Env column, while EQA/VLA use plain-text status glyphs (✅/❓/❌/🅼/⏳/▫); status-badge vocabulary, column schemas (Method · Env · Status · Graph/roadmap), and Part-I section ordering (env / tool / method / datasets / eval-infra / gaps) are not yet uniform. Standardize the three onto one shared shape — likely a .claude/standard/support-status-style.md defining the column conventions + a single badge legend — then normalize all three pages (including VLN) against it.

Drive the remaining {vln,eqa,vla}/unverified/ graphs to a trustworthy end-to-end datapoint (SR/SPL/LLM-Match + run_id) and promote the ones that pass to verified/, the way voxposer_libero_decomposed went (2026-06-30, libero_object 0.88). Overlaps the gym-migration re-verification wave (#64: smartway_ce, smartway_mono_ce, navgpt_ce, opennav_habitat, straightforward, + mp3d/simpler) and the VoxPoser/Pi-0 blockers (#65, #66). Progress 2026-07-15: verified/ now holds 13 graphs — vln 7 (incl. threestepnav_ce, promoted since), eqa 4, vla 2. Still pending a datapoint: discussnav_mp3d (100-ep faithful-config run; the 5-ep smoke is SR 0.2), tooleqa_hmeqa (also gated on #63), vla_policy_libero/_simpler (#66), the two pyslam_* probe graphs; blocked on port drift: navgpt_ce, opennav_habitat (#64), spatialnav_mp3d (M13, SSG CSV). Track progress against the §1 support matrices on the three support-status pages.

Continuation of the 2026-06-29 policy_adapter_vlnce (then policy_vlnce) refactor (env→canonical standardizes only; instruction tokenize + obs_transforms moved to canonical_to_model; _full_raw_obs leak removed; tokenize_instruction node retired; verified 3-ep bit-identical to baseline). Remaining: (1) explicit typed rgb/depth/instruction ports at the env→adapter boundary — retire the raw_obs ANY passthrough and rewire the obs iterIn/iterOut loop; (2) land the unified canonical.py v0 (flat schema + plain-dict info + action_space; scratchpad draft) as a shared policy module and migrate policy_adapter_vla onto it; (3) verify the RxR / feat path (rxrce + Seq2Seq) — aligned but not eval'd; (4) carry the standardize/process split into policy_adapter_vla. Mental model + done-state in policy-adapters.html §1.2/§8.