Blueprint

§1Problem Statement

The lede above says what AgentCanvas is; this section is the why. Skip to §2 Capabilities for what it does.

Embodied agents — spanning VLN (Vision-and-Language Navigation), EQA (Embodied Question Answering), and VLA (Vision-Language-Action) — are increasingly built by composing foundation models with perception, mapping, memory, planning, and action. Unlike end-to-end policies, whose structure is absorbed into weights, this architecture is explicit and editable. That raises the question AgentCanvas is built around — can agent design be searched rather than hand-built? — alongside two stacks of pain it has to clear on the way.

Agent architecture is hand-built — and could be searched

Each agent fixes a choice at every join — sensor abstractions, map representations, memory state, prompt structure, planner topology, model placement, action interfaces — by hand, usually for a single benchmark. As foundation models and embodied tools multiply, the space grows faster than manual iteration can cover, so the natural move is to search it rather than hand-tune it.

Agent Architecture Search (AAS) already does this for text-domain agents, but embodied transfer is not free: stateful simulators, noisy multi-episode scoring, long perception/action traces, and no off-the-shelf palette of embodied primitives. AgentCanvas is our attempt to supply the missing substrate — a scaffold a coding agent can read, edit, run, and verify — so that searching agent design becomes possible for embodied agents too.

Embodied-specific pain points

• Modern embodied stack is thick — a working embodied agent needs LLM reasoning + tool use + simulator coupling + spatial tools, all wired together. Building this from scratch per project is prohibitively costly, and most of the effort goes into the execution layer rather than the idea being tested.
• Engineering nightmare — an embodied agent is not one model but a whole system — a stateful simulator plus a stack of heavy models and tools. Just running it, let alone at the scale benchmarking needs, is a hard engineering job in itself:
  • Python env hell — no single Python env satisfies every part; each simulator, VLM, detector, and policy pins its own clashing CUDA / torch / Python, so finding one runtime they all share is often impossible — you end up maintaining several incompatible environments just to load the agent.
  • Batching — each worker's simulator is a separate stateful process advancing at its own pace; you can batch the model but not the sims, so every step becomes an async gather-observations → batch-infer → scatter-actions dance.
  • Other infra — multimodal trajectories that must be logged and replayable, checkpoint/resume of multi-hour GPU runs that will crash, and debugging across process boundaries.
  Over a single paper's research cycle, the researcher pays far too much of this engineering cost instead of focusing on the algorithm itself.
• Hidden ground-truth dependencies — many methods quietly rely on simulator-provided ground truth (object poses, semantic labels, navigability) rather than real perception. Sometimes that's a legitimate way to control the experiment — but, oversight or not, it often goes unmentioned in the paper.

Common AI research pain points (amplified here)

• Non-reproducible implementations — every paper builds its agent from scratch with a different codebase; comparing methods fairly or reproducing results is painful — and many of them are Code coming SOON (Someday, Or Obviously Never).
• Paper ≠ code — papers show clean flow diagrams, but the actual code diverges in undocumented ways. Reproducing a paper means reverse-engineering its implementation.
• Tightly coupled code — domain logic (prompts, tools, policies) is tangled with infrastructure. Swapping one component means rewriting the pipeline.

What the platform has to demonstrate as built & shipped

• 110+ node types — 14 built-in plus ~100 domain types auto-discovered from workspace/ at startup; new types drop-in via Python files without framework edits.
• Dataflow executor handling cyclic agent loops natively via IterIn/IterOut gates, with playback controls (play / pause / step / stop).
• Server-mode nodesets running in their own Python interpreters (Habitat 0.1.7 in ac-vlnce env, MatterSim in ac-mp3d), auto-generated proxy nodes from declared port surfaces.
• Real-time WebSocket streaming of observations / reasoning / actions / metrics, routed by execution_id for concurrent runs.
• Multi-episode batch eval harness (POST /api/eval/v2/start → backend-owned JobScheduler spawns a subprocess-per-run BatchEvalRunner + EnvWorkerPool) replacing the retired vlnworkspace.eval Hydra CLI.
• Hot reload (POST /api/components/reload) for new node types without server restart.

§2Capabilities

AgentCanvas is a visual agent design platform where researchers rapidly prototype embodied AI agents by drawing node graphs on a canvas. Graphs execute in real-time against simulator environments (Habitat, MatterSim/MP3D, SIMPLER, LIBERO) or real-world setups — no pipeline code required.

Nine capabilities make this work. Each card opens its implementation detail.

§3Design Principles

1. Framework has zero domain knowledge agentcanvas/backend/app/ is a pure execution engine. All domain-specific code (policies, prompts, tools, environments) lives in workspace/. The framework never imports domain code directly — it discovers and bridges components at runtime.
2. Graph, not script Agent behavior is defined as a dataflow graph, not imperative code. This makes architectures visual, composable, and inspectable. The graph is the source of truth — not a visualization of code, but the code itself.
3. Dataflow execution Nodes fire when their inputs arrive, not in a fixed order. Naturally supports cyclic agent loops (observe → think → act → repeat) and enables future parallelism and branching.
4. Composability via nesting Composites let researchers group nodes into reusable sub-graphs at any depth. Execution always flattens to a single dataflow graph — no outer/inner split at runtime. Keeps the executor simple while enabling arbitrarily deep hierarchical agent architectures.
5. Real-time observability Every step streams observations, reasoning, actions, and metrics via WebSocket. Researchers see what the agent sees, read what it thinks, and can pause/step/resume at any point.
6. Extend, don't fork New capabilities are added by dropping Python files into workspace/, not by modifying framework code. The component system auto-discovers new tools, skills, agents, environments, policies, and node handlers.

§4Core Components

§5Core Features

Visual agent composition

• Unified canvas paradigm (ADR-canvas-002): one editor, one node registry (110+ types), one catalog at every graph level — root canvas and subgraphs use identical tools
• Flat workspace (ADR-canvas-001): all node types on one canvas — no outer/inner layer split
• Drag-drop node editor with optional recursive composite nodes (ADR-dataflow-001)
• Pre-built graph templates (Straightforward CMA, NavGPT-CE) — loaded as flat workspace JSON
• Type-checked wiring with 9 wire types and color-coded handles
• Unified sidebar catalog: Environment, LLM, Prompt, Loop Control, Output, Composite, Agent Presets, Saved Graphs
• Graph save/load: two kinds — Graphs (editable templates in workspace/graphs/, open in tabs) and Graph Nodes (frozen composites in workspace/graph_nodes/, drag onto canvas as reusable blocks with snapshot semantics). Explorer sidebar shows both as separate sections with VS Code-style tree expansion (ADR-canvas-003)
• Composite nodes: optionally group nodes into reusable composites with graphIn/graphOut boundaries
• Execution toolbar: Play/Pause/Stop/Step/Reset controls at page level (no AgentLoop wrapper needed)

NodeSet manager

• Dedicated page for managing NodeSets, server-mode nodesets, and environments — separate from canvas
• Load/unload NodeSets: activation makes tools appear in canvas sidebar, deactivation removes them
• Start/stop/restart server-mode nodesets (Habitat, MatterSim, SIMPLER, etc.)
• Rescan workspace/ for new components
• Auto-hosted deployment (ADR-server-001): any BaseNodeSet can be loaded in server mode — framework auto-generates the server from tool port definitions, zero extra code

Real-time execution & streaming

• Single /run endpoint: send graph definition + execution_id, get WebSocket events
• 8 WebSocket event types: nav_step, nav_llm_step, nav_status, nav_complete, and more
• Playback controls: play, pause, resume, step, stop
• execution_id routing: multiple concurrent agent runs with isolated output streams

Iteration & cycles

• Two-sided IterIn/IterOut gates for clean cyclic graphs (no back-edges) — IterIn captures run-start seeds on its left side; IterOut carries loop state and a stop input that halts the loop
• Configurable step budget (default 500)
• Loop ends when the IterOut stop port goes true, the budget is exhausted, or the user stops — the verdict stage runs once on the after-loop band (the former Termination node was removed; see glossary §6)

Extensible component system

• 2 base classes: BaseCanvasNode, BaseNodeSet
• Auto-discovery: drop a Python file in workspace/, reload, it's available
• Lifecycle hooks: async initialize() / shutdown()
• Hot reload: POST /api/components/reload
• Saved composite graphs via /api/graphs (CRUD for workspace/graphs/*.json)

State container system (ADR-dataflow-002)

• Dual-wire architecture: data edges (typed dataflow) + state edges (access grants to shared state containers) coexist on canvas
• StateContainer: dict of named BaseState entries — each with a reducer (accumulator, lastWrite, counter), a value type from an extensible registry, and a lifetime field (forever / step / episode / run / custom) that subscribes the state to framework signals for declarative clearing
• State value types: extensible registry — TEXT, BOOL, METRICS, POSE plus domain types (POINTCLOUD, OCCUPANCY_MAP, EMBEDDING, EPISODE_CONTEXT) and ANY
• State edges: separate wire system — dashed violet lines granting read/write access, toggled via toolbar button, __state__ port on every node
• graph_state: optional graph-level container auto-accessible by all nodes — displayed in State tab of bottom panel
• Node decomposition: "fancy" method nodes (e.g. a history tracker or a reason step) can be decomposed into composite graphs with visible internal state
• Checkpointable: each state implements checkpoint/restore for persistence and replay

VLN-CE evaluation

• Native path: Habitat policies with .act() interface
• External path: custom models with .predict_action() interface
• Metrics: SPL (primary), SR, nDTW, SDTW + per-episode breakdown

§6Differentiation

AgentCanvas combines three structural features that are collectively absent from current agent frameworks:

The triple matters because it makes the platform's content (graphs) structurally inspectable: graph topology is a JSON artefact, port types are declarable and checkable, and tool boundaries are explicit nodeset surfaces. This is what makes AgentCanvas a viable substrate for downstream automation — for example, an automated faithfulness audit can reduce its rules to JSON-checkable structural questions only because of these surfaces. Without all three, audit cost dominates run cost.

§7Open Questions

• Streaming graph mutations — runtime in-graph mutations (add/remove/rewire nodes mid-execution) vs current snapshot-and-reload model. Latter is simpler and what the AAS architecture-search loop currently uses; former unlocks self-modifying agents but multiplies executor complexity.
• State container reducer DSL — Python callable per reducer (current) vs declarative DSL (e.g. JSON spec → reducer factory). Latter would let reducers ride in graph JSON for full self-containment; former is more flexible.
• Multi-tenant execution — single global executor (current; single-user assumption) vs per-user executor pool with isolation. Becomes load-bearing only if AgentCanvas ships as a hosted service.
• Graph node snapshot semantics — copy-on-save (current; immutable graph nodes) vs reference (live updates propagate from source graph). Current is safer; reference would feel more git-like and reduce stale-snapshot drift.
• graphIn/graphOut arity — fixed 1-port boundary (current) vs N-port boundary (multiple inputs/outputs per composite). Fixed simplifies the flatten algorithm; N-port matches researcher mental model better.

§8Status

Core platform stable: graph executor, IterIn/IterOut iteration, 110+ node types, server-mode nodesets (Habitat / SIMPLER / Prismatic), WebSocket streaming, NodeSet Manager, hook system. Bottleneck: Eval harness frontend (Feature TODO F5) is incomplete — backend works but the Eval page in agentcanvas/frontend/src/eval/ still has gaps. Runtime graph mutation — letting the AAS search loop edit graphs through typed operations rather than ad-hoc file rewrites — is the headline v2 (Topology-Growing Canvas) capability, out of scope for the v1 bounded-topology paradigm. Next: finish F5 frontend.