Roadmap

First monolith smoke (3 ep) exposed a metric/termination gap in tooleqa__step + workspace/graphs/eqa/unverified/tooleqa_hmeqa.json: episodes where the 3B agent never calls final_answer run to the env step-limit and emit no metrics port, so they're silently excluded from the SR aggregate (harness reported 1.0 = 1/1 metric-producing episode, true result 1/3). Three fixes: (1) the max-iter fallback keys off env step_index, which only advances on go_next — switch it to the ReAct step count so non-terminating episodes still emit a (failed) answer + metrics; (2) metrics.num_steps reports 0 — populate with the ReAct step count; (3) count env-done-without-answer episodes as success=0 rather than dropping them — partially covered since 2026-06-11: the batch runner now flips an eval episode with no graphOut snapshot to status=error (no more silent exclusion), but emitting an explicit failed verdict is still the right fix. Also investigate why the 3B keeps exploring instead of answering (prompt/format adherence vs upstream 7B).

Nodes + graphs exist (workspace/nodesets/method/navgpt.py + navgpt_mp3d_tools.py, graph JSONs); remaining: agent manifest, skills module, consolidate two graph variants, clean organization (merged with former #27).

Verify the functionality of our future state container system — identify a published agent architecture whose reasoning loop requires shared persistent state across steps. Implement as graph preset using state containers to validate the dual-wire design before broader rollout.

Support multiple Habitat-Sim versions, adapt nodesets/policies for different VLN-CE releases and dataset formats.

installation.md rescued to docs/getting-started-installation-rescued.md (131 lines); remaining: move to active docs, update and verify.

Low priority. LoopRunner is 115 lines with clean separation; GraphExecutor well-structured. Revisit only if bugs surface in IterIn/IterOut, pause/resume/stop lifecycle.

in-memory checkpoint wiring at IterOut (done), checkpoint/restore API (done), migrate HistoryTrackerNode/LLMCallNode/NavGPTReasonNode to graph_state (done), NavGPT-CE graph JSON with graph_state (done). Remaining: decompose HistoryTracker into composite graph with visible state, add state config panel for editing container states in the frontend, implement container checkpoint persistence to disk.

Current frontend logic is fully Claude-generated; audit canvas, API client, store, and node types against actual backend endpoints and models to fix any drift or assumptions.

Unclear scope — define concrete deliverables or remove.

Audit and reorganize the top-level directory layout for clarity and long-term scalability: review workspace/ subdirectory conventions (agent/, nodes/, nodesets/, graphs/, graph_nodes/, skills/, servers/, policies/), evaluate whether agentcanvas/backend/app/ structure scales, align folder naming with documented concepts. Define a canonical layout doc.

Subdivide NodeSets into finer categories (env, ML model, reasoning/tool) and write per-category contributor guide covering scaffolding, registration, UI config, server-mode considerations.

Custom method nodes are proliferating (NavGPT/MapGPT/DiscussNav each ship their own parse/format/reason trio), making the canvas look like ordinary visualization over one-off code. Framework abstractions are solid; the domain layer is thin. Candidates: parameterized reasoning turn (prompt template + LLM + structured parser), history formatter with pluggable strategies (full / last-k / summarized), action-space adapter (discrete panoramic ↔ continuous waypoints), observation encoder (pose + RGB + depth → text/image bundle). Do the extraction pass against NavGPT + MapGPT + DiscussNav + VLN-Zero concretely.

Follow-up to #10. Every node in habitat.py, sam.py, policy_cma.py, others.py currently only sets NodeUIConfig(color="..."). Audit which nodes should expose inline editable config_fields and display_fields. Per node: color-only is fine, or upgrade to block/strip/viewer layout. One-line classification per node, then edits for ones that need the richer form.

Different env nodesets expose radically different action ports today (MP3D: dynamic-cardinality discrete viewpoints via viewpoint_id TEXT + cand_vpids_json; Habitat-continuous: 2-vector or Discrete(4) ACTION int; VLN-CE: discrete Discrete(4); future AI2-THOR: parameterized actions with object IDs), which forces every agent graph to be env-specific. Proposal (deferred): every env's reset/step emits a single action_manifest (TEXT, JSON-serialized) with a first-field type discriminator describing the currently valid action set. For now: keep env-native shapes; revisit once a third env nodeset lands and we can see the real minimal common schema.

Exercise all 4 phases (Port → Trace → Nodeset → Graph) on a fresh method to stress-test tutorials + shims.

Audit docs/pages/developer-guide/core/glossary.html for stale/duplicate/legacy entries, trim verbose definitions, align terms with current ADR state.

UI buttons unresponsive (clicks have no effect); investigate event handlers / API wiring.

Today errors are silent on two paths: (a) server-mode nodes' _self_log("error", ...) is not carried back over the proxy HTTP response; (b) many built-in tools swallow soft failures into empty default outputs without exposing an error output port.

Scope TBD (likely covers workspace/{graphs,graph_nodes,nodesets,policies} discovery, outputs/eval_runs/ vs data/outputs/{eval,eval_mock,eval_native}/ vs legacy outputs/runs/ JSONL persistence, overlapping ownership with TODO #40 — clarify what "file management" covers before scheduling).

Under worker_count > 1 with a batched=True node hosted in a parallelism="shared" server-mode subprocess, _BatchQueue._delayed_flush raises RuntimeError: got Future attached to a different loop (observed 2026-05-07 on policy_vlnce__predict at K=10; 88/100 episodes returned metrics={}). Workaround in tree: policy_adapter_vlnce__predict.batched = False (node renamed from policy_vlnce__predict in the 2026-07-04 split). Real fix: make _BatchQueue loop-agnostic or per-worker.

Current architecture erases composite at compile time via flatten_graph — leaky abstraction. Cleaner: composite stays as a real runtime entity, executor walks composite boundaries natively. Trade-off: flatten = simple executor + complex compile-time transform with leaky patches; composite-native = composite-aware executor + simple boundary resolution. Trigger: third leak shows up, OR multi-scope authoring becomes the dominant pattern. Until then, the conditional-strip middle option keeps things working.

13 install_*.sh in scripts/install/ plus 11 conda envs on disk. Naming inconsistent (agentcanvas-X vs bare vlnce/hmeqa/vlaws*), env→nodeset mapping implicit, several install scripts duplicate boilerplate. Audit goals: document canonical env↔nodeset table; decide which envs can merge; extract shared install boilerplate; clarify install_all variants or collapse them; align naming.

Each v1 agent form should have its own tutorial page under docs/pages/developer-guide/tutorials/ showing a minimal worked example: DAG / cyclic loop / ReAct hidden / ReAct router / Plan-and-Execute / bounded multi-agent / embodied sense-act / FM pipeline. Canonical list lives in major-versions.html §1 "Agent forms v1 covers".

Opening a graph synchronously activates its nodesets. The frontend now shows a loading overlay + Cancel, but Cancel only aborts the client fetch — it does not stop the backend load. True hard-cancel is only possible at the process boundary = server mode: a live GPU/CUDA object's ownership is bound to the process that created it and cannot be handed back to the parent, so "load in a subprocess then return the object" fails for GPU models (works only for pure-CPU picklable objects, which rarely load slowly enough to need interrupting). Plan: (a) offload the synchronous BaseServer.start() to a thread, fixing the framework hole where server-mode startup blocks the entire event loop for up to startup_timeout (1800s); (b) run the ensure endpoint as a token-keyed asyncio.Task + add POST /nodesets/ensure/cancel, where server-mode cancel kills the subprocess; (c) auto-promote heavy local nodesets to server mode so they gain hard-cancel; lightweight local stays on client-side fetch abort (soft cancel).

First full rewrite of docs/pages/developer-guide/design-docs/graph/graph-executor.html landed 2026-06-08 (Part I concept SVGs + flowchart; Part II verbatim run()/_fire_node/helpers with ◀── annotations). Committed as a checkpoint — refine later: tighten prose, check dark-mode contrast and flowchart arrow routing in-browser, consider trimming the full-verbatim listings if they prove too long to maintain. 2026-06-11 (evening): the listing problem is structurally resolved — Part I split into graph-view/engine-view, §4.5 redrawn around the ready-queue, §4.6 now a turn-by-turn table; Part II slimmed to run() only (_fire_node/helpers read in source) and the listing is synced from source by docs/_lib/_sync_run_listing.py (comments are the annotations; Pygments two-stage highlighting per .claude/standard/code-highlighting.md). Remaining: in-browser pass — dark-mode contrast, flowchart arrow routing, the new GitHub-palette listing.

After the env-nodeset gym-like refactor (2026-06-09), the per-step evaluate node (triggered by step_*.info) does not fire after the agent's final STOP step — the loop terminates (e.g. parse_response.is_stop → termination) before evaluate re-fires — so the harvested output_port__metrics graphOut holds the pre-STOP value (success=0). Confirmed on smartway_ce (20 ep, gpt-5-mini, step_budget=15, run 20260610_091657): habitat computes success=1.0 on the STOP step (visible in step.info.metrics) but 5/20 episodes that stopped within the 3 m radius (ep 9/10/11/14/17, dist 0.43–2.61 m) recorded success=0 → reported SR 0.000 instead of ≈0.25 (baseline 0.28). Mechanism, not agent regression (nDTW 0.50 ≥ baseline 0.40; oracle 0.40). Structural fix landed 2026-06-11 (two-sided iterOut refactor): the termination node was removed, evaluate moved to the after-loop band fed once at termination via the iterOut's final_stop handle, and the validator now rejects loop-body-fed metrics/success graphOuts — the engine-level regression test asserts the after-loop consumer receives the terminal iteration's value. Remaining work = re-verification: smartway_ce (re-verified — promoted to vln/verified/), straightforward (re-verified 2026-06-28: bit-identical to native CMA, 50-ep SR 0.38 / SPL 0.348 — promoted to vln/verified/); 2026-07-05 wave: smartway_mono_ce (interface-equivalence audit — env wiring handle-identical to re-verified smartway_ce), navgpt_mp3d (dead legacy state edges removed; clean 1-ep smoke 20260705_194214, full metric harvest via final_stop), octo_simpler (clean 1-ep smoke 20260705_193758) — all promoted to verified/; still open: navgpt_ce (llmCall action/summary port drift), opennav_habitat (llmCall template drift), + the remaining mp3d/simpler graphs. See env template.

Surfaced 2026-06-11 while smoke-testing the gym-migrated vla_policy_libero.json. The graph's ✅-verified datapoint (2026-05-02, 5/5) came from scripts/smoke/smoke_vla_libero.py against the external vlaworkspace; the canvas-graph path through the policy_adapter_vla server has been broken since the 2026-05-04 adapter split. Debt chain, outermost first — first three fixed 2026-06-11: (1) graph checkpoint_path pointed at a deleted vlaworkspace slurm dir → repointed to vendored data/vla_policy/checkpoints/pi0_libero_pytorch; (2) LiberoRobot lost its RobotAdaptor base in the adapter split, so ensure_robot's _find_subclass never matched → re-based; (3) adapters/models/pi0_model.py kept a one-level-short relative import (..models.openpi → resolves to adapters.models.openpi) → ...models.openpi; (3b) ac-vla-policy env had pip nvidia-nvjitlink-cu12 12.5 vs 12.9-era CUDA libs → upgraded to ~=12.9. Remaining blocker: Pi0Policy.__init__ with DEFAULT_KWARGS use_pretrained_weight=True tries a JAX→PyTorch conversion via python -m vlaworkspace.model.convert_jax_model_to_pytorch (module not vendored; cache_dir 'data/models' also doesn't match the vendored layout data/vla_policy/checkpoints/pi0_base_pytorch) — even though the finetuned safetensors is already local. Fix = make the pretrained-base path resolve to the vendored pi0_base_pytorch (or skip base-init when a full finetune checkpoint is given), then smoke vla_policy_libero 1 ep. Companion engine hole (cousin of the silent-episode-completion memory): a server proxy node whose response is {"x": null} gets logged as outputs={} with error=null and the episode "completes" at step_count=0 — three of today's four failures were silent for this reason.

A follow-up cleanup that surfaced from the #67 transport work. env_habitat's panorama path (observe_panorama, env_habitat.py:536-537) emits two depth products per view: depth_base64 (8-bit min-max-normalised PNG via encode_depth_base64, viewer-only, lossy / relative) and depth_raw_base64 (16-bit-millimetre PNG via encode_depth_raw_base64 env_habitat.py:798, metric-faithful, capped at 65.535 m to fit uint16). This base64-PNG encoding is an application-layer contract consumed by downstream method nodesets (opennav_perception._decode_depth_raw, aoplanner._decode_depth_m, smartway_waypoint) — distinct from, and predating, the Move-1 msgpack transport (#67), which can now carry a raw ndarray across the env boundary as a blob ExtType with no base64 at all. Not a bug — kept intentionally because (1) the consumers' wire contract is a base64 string; (2) 16-bit PNG compresses the depth map (a raw float32 480×640 panorama set is large); (3) the "16-bit / mm / cap 65.535 m" metric semantic is carried by the I;16 PNG format choice; (4) it survives the JSON legacy path too. Cleanup question: once the JSON legacy path is retired (transport msgpack-only) and the consumers can be migrated together, depth_raw_base64 could collapse to a raw depth ndarray + a units flag (is_mm/metres), letting msgpack own compression/encoding and removing the bespoke encode_depth_raw_base64 / _decode_depth_raw pair. Scope: weigh the bandwidth win of PNG compression against the simplification before scheduling; relates to #67 (transport contract) and the viewer-vs-metric depth split.

Scale batch eval from a single machine to coordinated multi-machine execution (an AgentCanvas open-source prerequisite, from the find-job dependency graph in the personal todo plan). Prerequisite: first map out the current batch-eval implementation (single-machine worker pool / JobScheduler status quo, the 8765–8769 port pool, admission control) before designing cross-machine scheduling.

Add runtime observability to the execution engine (an AgentCanvas open-source prerequisite, from the find-job dependency graph): collect per-node execution time, inter-node data-transfer (wire/transport) latency, etc., to locate bottlenecks.

The three support-status pages (vln-/eqa-/vla-support-status.html) drifted while being converted onto the VLN two-part template (2026-06-30). Divergences to reconcile: VLN's §1 matrix uses CSS-styled ours-* badge spans plus an Env column, while EQA/VLA use plain-text status glyphs (✅/❓/❌/🅼/⏳/▫); status-badge vocabulary, column schemas (Method · Env · Status · Graph/roadmap), and Part-I section ordering (env / tool / method / datasets / eval-infra / gaps) are not yet uniform. Standardize the three onto one shared shape — likely a .claude/standard/support-status-style.md defining the column conventions + a single badge legend — then normalize all three pages (including VLN) against it.

Drive the remaining {vln,eqa,vla}/unverified/ graphs to a trustworthy end-to-end datapoint (SR/SPL/LLM-Match + run_id) and promote the ones that pass to verified/, the way voxposer_libero_decomposed went (2026-06-30, libero_object 0.88). Overlaps the gym-migration re-verification wave (#64: smartway_ce, smartway_mono_ce, navgpt_ce, opennav_habitat, straightforward, + mp3d/simpler) and the VoxPoser/Pi-0 blockers (#65, #66). Progress 2026-07-05: smartway_mono_ce, navgpt_mp3d, octo_simpler promoted (see #64); the octo_simpler verified-status discrepancy is resolved (promoted after a clean 1-ep smoke). Still pending a datapoint: discussnav_mp3d (interface-clean, fitness not paper-comparable), tooleqa_hmeqa, vla_policy_libero/_simpler; blocked on port drift: navgpt_ce, opennav_habitat, spatialnav_mp3d. Track progress against the §1 support matrices on the three support-status pages.

Continuation of the 2026-06-29 policy_adapter_vlnce (then policy_vlnce) refactor (env→canonical standardizes only; instruction tokenize + obs_transforms moved to canonical_to_model; _full_raw_obs leak removed; tokenize_instruction node retired; verified 3-ep bit-identical to baseline). Remaining: (1) explicit typed rgb/depth/instruction ports at the env→adapter boundary — retire the raw_obs ANY passthrough and rewire the obs iterIn/iterOut loop; (2) land the unified canonical.py v0 (flat schema + plain-dict info + action_space; scratchpad draft) as a shared policy module and migrate policy_adapter_vla onto it; (3) verify the RxR / feat path (rxrce + Seq2Seq) — aligned but not eval'd; (4) carry the standardize/process split into policy_adapter_vla. Mental model + done-state in policy-adapters.html §1.2/§8.