Roadmap — Done — AgentCanvas

Split out of roadmap so the main roadmap stays focused on open work. This file is the historical record. Read the per-field ADRs for the architectural reasoning behind the ADR-labelled items below. Note: entry names are ship-time snapshots — some have since been renamed or removed (e.g. the "DAG executor", the "Ephemeral" reducer, BatchEvalOrchestrator); the glossary is the source of truth for current names.

§1Done — Shipped Features

Two-layer canvas architecture (outer graph + inner graph editor)
Dataflow executor with IterIn/IterOut iteration gates
DAG executor for acyclic graphs
Component system with auto-discovery from workspace/
WebSocket streaming with execution_id routing (8 event types)
Single /run endpoint with pause/resume/stop lifecycle
Graph presets: Straightforward (CMA), NavGPT-CE
Wire type system: IMAGE, DEPTH, ACTION, STATE, TEXT, BOOL, METRICS, OBSERVATION
Hydra eval harness with native + external policy paths
MkDocs Material documentation site
Typed graph model (GraphDefinition, NodeDef, EdgeDef in graph_def.py) — backend validation, JSON file management, CRUD API
AutoServerApp: write-once deploy-anywhere for nodesets (ADR-server-001)
Unified canvas paradigm (ADR-canvas-002) — one editor, one registry, one catalog at every graph level
NodeSet Manager page (ADR-canvas-001) — dedicated UI for nodeset load/unload, server start/stop/restart
BaseEnv removed — environments are NodeSets (ADR-components-002)
Standardized node types nodeset__node convention, auto-load nodesets on graph open (ADR-components-003)
Legacy two-layer files deleted (nodeTypes.ts, innerNodeTypes.ts, FlowCanvas.tsx, InnerGraphEditor.tsx, AgentLoopCanvasNode.tsx, …)
Python-driven node UI (NodeUIConfig + ConfigField on BaseCanvasNode) — all node rendering driven from Python, GenericBlockRenderer handles block + strip layouts
Mock/legacy node cleanup — deleted 23 mock .tsx files, all mock nodes labelled [Mock] in sidebar + canvas
Custom .tsx reduced from 23 → 10 (6 custom UI + 1 renderer + 2 shared + 1 composite helper)
State container system (dual-wire architecture): StateContainer as dict of named BaseState entries, 4 reducer types (Accumulator, LastWrite, Counter, Ephemeral), extensible value type registry, state edges as separate wire system, canvas toggle, executor integration, flatten support
Graph node system (ADR-canvas-003): kind field on GraphDefinition, dual storage (graphs/ + graph_nodes/), Explorer split into Graphs + Graph Nodes sections, VS Code-style tree expansion, "Save as Node" button, snapshot semantics
Auto-layout system: swim-lane layout algorithm (app/layout.py), POST /api/graphs/layout endpoint, Auto Layout button. Updated 2026-04-19 for ADR-dataflow-006 three-pivot: nodes split into init band + loop band stacked vertically with BAND_GAP
Per-node LLM/VLM model selection via profile config field in LLMCallNode
LLM config system redesign (ADR-platform-003): layered field-level env var fallback (profile > AGENTCANVAS_* > VLM_* > registry), argparse CLI (8 commands, works without server), mtime-based ProfileStore cache invalidation, key validation, interactive setup script
VS Code-like UI layout: ExplorerPanel (tree sidebar), TabBar (tabbed multi-graph editing), OutputDrawer
BaseSkill removed — skills refactored to BaseCanvasNode subclasses in vln_skills nodeset
agentcanvas general-purpose env (ADR-platform-004): agentcanvas (Python 3.10+, modern torch/transformers + FastAPI) as THE environment for all development, research, and runtime; vlnce (Python 3.8) retained only for the habitat-sim 0.1.7 binary constraint. Framework import guard test enforces the boundary.
ADR-components-004 recorded for Python-driven node UI
NodeUIConfig documentation with examples in capabilities/customizable-node-system.md
Eval page redesign (ADR-eval-001): graph-driven batch evaluation via BatchEvalOrchestrator wrapping LoopRunner, ExecutionPrinciples for batch mode, ExecutionGuard for exclusive canvas/eval lock, env metadata interface on BaseNodeSet, registry-backed graph introspection, shared nodeset auto-loading, v2 API at /api/eval/v2, new frontend (9 components)
Node output log system (ADR-observability-003): two-layer execution logging — automatic exterior (executor captures port I/O + timing per node firing) + voluntary interior (_self_log() on BaseCanvasNode). ExecutionLogger with JSONL persistence. log_serialize() truncates large values. exec_log WS events (suppressed in eval mode). REST API at /api/logs with 4 GET endpoints. Frontend LogPanel in OutputDrawer Logs tab.
Canvas environment & episode control (ADR-legacy-002): standalone EnvController Protocol. HabitatEnvController delegates to HabitatEnvManager.get(). REST API at /api/env/{name}/ (5 endpoints). POST endpoints guarded by ExecutionGuard. Pre-flight pattern: frontend calls envSetEpisode() before runPipeline() — keeps RunRequest/LoopRunner env-agnostic. EnvContextBar component, auto-advance on episode completion.
Configurable IterIn/IterOut ports (ADR-dataflow-003): config.ports on NodeDef overrides class-level ports per instance. BaseCanvasNode._resolve_ports(config) classmethod. Domain ports removed from framework class defaults. Frontend resolveInstancePorts() utility, PortListEditor component. All 3 graph presets migrated with explicit config.ports.
LIST[T] wire-type modifier (ADR-dataflow-005): opt-in list modifier with consumer-side coercion. LLMCallNode.rgb migrated to LIST[IMAGE], unblocking MapGPT/MC-GPT/DiscussNav/VLN-Zero multi-image prompts. graph_def.validate_edge_wire_type compatibility helper; frontend PortListEditor checkbox modifier + isCompatibleWireConnection + doubled-ring handle visuals.
Viewer system refactor: per-instance viewer data via viewer_data WS events keyed by node_id. _SinkBase.forward() serializes wired inputs via serialize_for_display(). Display type registry (viewerFieldRegistry.ts) with lazy-loaded field renderers. DisplayField.accumulate for log-type fields. 5 viewer sinks: ImageViewer, ThinkingLog, ActionLog, Metrics, TextViewer.
Configurable imageViewer replaces fixed observationViewer (ADR-components-007): new ImageViewerSink with _resolve_ports deriving ports from config.ports (IMAGE/DEPTH only) + rows/cols grid; new imageGrid frontend layout + ImageGridViewerLayout.tsx; 4 workspace graphs migrated.
Hybrid data-layout unification (ADR-platform-005): data/ restructured into habitat/ (VLN-CE framework mirror) + mp3d/v1/scans/ (asset-kind namespace) + opennav/ (tool namespace) + outputs/{eval, eval_mock, eval_native}. New scripts/data/migrate_data_layout.sh (atomic rename(2), --dry-run, idempotent), scripts/data/gen_skybox_rgb_mp3d.sh (inline cv2 merger), scripts/smoke/vlnce_smoke.py. Bugs fixed: matterport3d.py:245 passed parent of v1/scans to setDatasetPath; MATTERPORT_DATA_DIR suffix match was unanchored.
Worker-pool batch eval + batched inference tier (ADR-eval-002): three-phase rollout — Phase A foundation (controller_overrides, EvalConfig.worker_count/per_step_budget_sec, BaseNodeSet.default_per_step_budget_sec, EnvWorkerPool seam at worker_count=1), Phase B multi-worker env fan-out (worker_count=N spawns N tagged subprocesses + per-worker URL routing + asyncio.gather over shared episode queue with results_lock), Phase C batched inference (BaseCanvasNode.batched/batch_dim ClassVars; BatchedInferenceServer + _BatchQueue + BatchedClient; policy_cma__forward opted in).
Graph SDK (shipped 2026-07-12): programmatic authoring + in-process running of the same GraphDefinition — Graph builder with loop/hook/composite sugar, catalog()/describe()/nodesets() introspection, typed authoring stubs (generate_node_stubs), load-time validation (GraphValidationError), run()/on_event RunEvent streaming + batch eval(), cassette record/replay for hermetic GPU-graph tests (graph_sdk_replay.py), graph_to_code() reverse codegen, and seven numbered example scripts (backend/examples/). See the capability page.
pySLAM streaming-SLAM nodeset (shipped 2026-07-10): model_pyslam — full SLAM stack in a GPL-isolated rootless-Docker container bridge (FastAPI shim in-container), canvas viewers, TUM demo graphs. Retires the old "SLAM integration — waiting on ORB-SLAM bindings" Planned item.
pip-installable SDK (shipped 2026-07-12): repo-root pyproject.toml packages app* + the agentcanvas public namespace as the agentcanvas distribution — stdlib-light core (pydantic-settings only), tiered extras [server] / [llm] / [backend] / [all]; test_dependency_boundary.py actively guards the light-core import boundary; per-node accounting ContextVars extracted to app/standard/telemetry.py.

§2Completed TODO Items

IDs are permanent, never reused. Full context in Done section and Changelog. A few entries are recorded here as closed — no longer valid (retired without shipping, e.g. superseded or scope-less) rather than completed; the ID stays reserved.

1 Doc-site: add glossary entries for NodeUIConfig, GenericBlockRenderer, ConfigField; document mock cleanup in relevant docs
2 Doc-site: update capabilities/customizable-node-system.md with NodeUIConfig examples for nodeset authors
3 Doc-site: update capabilities/real-time-observability.md — /node-schemas response now includes ui_config
4 Doc-site: record ADR for Python-driven node UI (NodeUIConfig replacing custom .tsx)
5 Dynamic sidebar catalog — show only tools from activated NodeSets + built-ins
7 Delete docs-site/docs/tmp_trash/ — deleted; installation.md rescued
8 Layout Component System (ADR-components-006): DisplayField, layout="viewer", refactor GenericBlockRenderer into dispatcher + layout components, eliminate 4 output viewer .tsx files
9 Verify saved graphs (Straightforward, NavGPT-CE) render correctly with GenericBlockRenderer
10 Add ui_config to nodeset tools — every node in habitat.py, sam.py, policy_cma.py, others.py has a NodeUIConfig ClassVar
12 Implement basic_agent_nodeset — general-purpose canvas nodes for LLM agents: note_write/note_read/note_list, web_search/web_fetch, image_analyze, measure_distance/compute_heading, get_instruction/get_step_count/get_history
15 Correct skills logic — BaseSkill removed; skills refactored to BaseCanvasNode subclasses
16 State container system (dual-wire architecture)
18 Better LLM/VLM call — per-node model selection via profile config field
20 Choose the actual running env for agentcanvas — ADR-platform-004
21 General UI & filesystem refactor — VS Code-like layout
22 Install instructions — getting-started/installation.html live on the doc-site; clean-room verified end-to-end on a fresh Ubuntu 20.04 host 2026-07-09 (install-verify fix waves: PRs #32/#34/#36)
24 Call for contribution — CONTRIBUTING.md, label good-first-issues
26 File explorer with sub-graph pull-down and graph node
30 Align skill and tool concepts with Claude Code patterns — closed, no longer valid: scope-less from the start (its own note said "define concrete deliverables or remove"), and the substantive work already happened elsewhere — BaseSkill was removed and skills refactored to BaseCanvasNode subclasses (see #15)
31 Running log system — two-layer node output logging (ADR-observability-003)
32 Check asynchronization system — closed, no longer valid: no deliverable was ever defined and the executor carries no async debt (no TODO/FIXME markers in graph_executor.py / loop_runner.py); the one real async fault, the BatchedInferenceServer cross-event-loop bug, is tracked precisely by #58
33 Unified proxy node types — auto_server_app.py:132,140 carries tool.node_type into manifest as FunctionSchema.name
35 Refactor episode control to use proper AutoServerApp protocol — ADR-server-002: replaced env-specific HabitatEnvController + 5 hidden tools with generic BaseController
38 Refactor viewer system — per-node viewer_data WS events, _SinkBase self-serializes
41 Document the pipeline for adding each kind of nodeset and node — per-type tutorials (.claude/tutorials/skill-canvas-node/-nodeset/-env-nodeset.md) + the component-creation and nodeset-layout standards + per-env doc-site pages (developer-guide/nodesets/)
42 env nodeset should have a setted pattern, need refactor
43 Figure out MP3D data story — resolved by ADR-platform-005: 90 skybox zips downloaded (18 GB), 10,912 panoramas across 90 scans
48 Validate /implement-graph pipeline end-to-end — exercised on multiple fresh methods (AO-Planner 2026-06-16, Three-Step 2026-07-03 via grill-implement, pySLAM 2026-07-08+), each reaching a runnable graph + verification datapoint
49 Clean up glossary — 2026-07-15 core-refine pass: terms aligned with current ADR state, Graph SDK section added, dead self-link removed, live filter implemented
19 Habitat versions & VLN-CE adaptation — resolved 2026-08-16: env_vlnce033 ("VLN-CE @ hs0.3.3") runs R2R-CE episodes on habitat-sim 0.3.3 without habitat-lab (faithful action/sensor caliber + in-nodeset NE/SR/SPL/OSR/nDTW), wired to CMA via the adapter pipeline (straightforward_vlnce033, 10-ep val_unseen SR 0.50); legacy env_habitat (0.1.7) stays frozen as the literature-comparable oracle
50 Decouple VLM backend from explore-eqa nodeset
51 Refactor auto-layout to handle the note built-in node
52 Make the Explorer panel resizable like VSCode — drag handle to adjust sidebar width
53 Debug Batch Evaluate page — start/stop fully wired to /api/eval/v2 with disabled-gating, every evalApi method maps to a real route, no dead handlers (audited 2026-07-15; the page was rebuilt in the ADR-eval-001 redesign)
56 Clarify method vs foundation-model boundary in nodeset design — resolved 2026-07-04: principle codified in the nodeset-layout standard and enforced codebase-wide (last method-embedded FMs extracted to generic wrappers: model_blip2, vlm_spatialbot, model_dinov2, model_ram); equivalence held by captured-input unit replay + per-graph 1-episode smokes
65 VoxPoser-LIBERO gripper convention inverted at the actuation boundary — fixed 2026-06-28 at exactly the env_libero__step_pose boundary (+ two more stacked convention bugs + grasp geometry); voxposer_libero_decomposed re-evaled and promoted to vla/verified/ (libero_object SR 0.88, 2026-06-30)
67 Server-mode transport contract — resolved 2026-06-15 with a msgpack codec replacing JSON+base64 (pack_body/unpack_body, blob ExtType for ndarray/torch/PIL with receiver-side degrade); formalized as ADR-server-004
68 Guardrail: stateful state-container on a shared server nodeset — resolved 2026-06-15 as a load-time warning (registry._check_container_ownership, escalatable to hard-reject); state_demo migrated to parallelism="replicated"
75 Harden ac_smartway install against relative editable paths — fixed 2026-07-09: -e third_party/... lines moved out of the env yaml into absolute-path install-script steps (mirroring the ac_vlnce fix); clean-room verified end-to-end on a fresh Ubuntu 20.04 host
71 System-log runtime statistics — per-node wall-clock in the Monitor page's Run view (services/system_runs.py + NodeTimingTable); per-node LLM usage and wire-transport accounting via the standard/telemetry.py ContextVars, read into each node's log entry
E1 Matterport3D Simulator nodeset — shipped 2026-04-18 as EnvMP3DNodeSet with three-field controller cascade (dataset → split → episode_index)
E2 RxR-CE dataset integration (Habitat-CE) — env_habitat loads RxR-CE via the upstream per-language YAMLs (en/hi/te), panel cascade dataset → split → episode_index, instruction language + extras surfaced on episode info; data at data/habitat/datasets/RxR_VLNCE_v0/
E5 Benchmark data pipeline — scripts/data/fetch_episodes_vln.sh takes --r2r / --r4r / --rxr / --rxr-full / --reverie / --reverie-features / --cvdn / --all / --dry-run / --force flags and lands 20 GB of episodes
E6 HM-EQA dataset/benchmark integration — completed 2026-05-09: workspace/nodesets/env/env_hmeqa/ (5 tools, free-pose teleport, ac-hmeqa env) + explore-eqa method nodesets; closed end-to-end on the 100-episode HM-EQA val standard split at SR 0.43 (≥ paper 0.38)
E7 RxR-CE BERT instruction features — 11,006 .npz (4.3 GB) at data/habitat/datasets/RxR_VLNCE_v0/text_features/, consumed by the CMA policy path (policy_adapter_vlnce canonical kind "feat") with rxr_cma_{en,hi,te} checkpoints in place
E9 OpenEQA (EM-EQA) integration — completed 2026-05-04: workspace/nodesets/env/env_openeqa_em.py (5 tools) + three graphs (blind-LLM / single-frame / multi-frame); first benchmark exercising free-text reasoning + LLM-as-judge eval
E11 SIMPLER nodeset — shipped 2026-05-01 as workspace/nodesets/server/simpler/ with reset/step/observe/metric/dataset_split/controller tools, paired with ac-simpler conda env (SAPIEN/ManiSkill2)
E12 LIBERO nodeset — shipped 2026-05-01 covering the four LIBERO task suites (Spatial / Object / Goal / 100), built on robosuite + MuJoCo via ac-libero conda env
E13 VLA Policy nodeset — shipped 2026-05-02 as workspace/nodesets/server/policy_vla/. Hosts Pi0 / SmolVLA / Diffusion Policy / DROID-DP via 4-node decomposition. End-to-end verified with Pi0 LIBERO finetune: 5/5 success, avg 83.8 steps, 5.4 s wall
M3 DiscussNav canvas graph — shipped 2026-06-16 (8-expert fan-out runs end-to-end; SR 0.2 on a 5-ep gpt-5-mini smoke). Still in vln/unverified/ — the 100-ep faithful-config datapoint + promotion are tracked by #73. (An earlier entry here wrongly carried M7's verification block.) PortBench v1 slot 3
M7 AO-Planner canvas graph — shipped 2026-06-16 as the faithful two-VLM VAP pipeline, verified at 100-ep scale 2026-07-04 (SR 0.35 vs paper zero-shot 25.5, not same-model); promoted to vln/verified/. PortBench v1 slot 4
M11 SmartWay canvas graph — waypoint-predictor + MLLM navigator with backtracking (smartway nodeset family); smartway_ce.json promoted to vln/verified/ after the #64 terminal-STOP fix, monolithic smartway_mono_ce.json promoted 2026-07-05 via interface-equivalence audit
M12 Three-Step Nav canvas graph — Open-Nav-family three-step pipeline ported as the threestepnav nodeset + graphs; faithfulness frozen by an 82-check upstream-import byte harness; 2026-07-03 aligned stack gives SR* 0.24 at n=25 (statistically compatible with paper 0.34); since promoted — threestepnav_ce.json sits in vln/verified/