Blueprint
What AgentCanvas is, the nine capabilities, design principles, and what's shipped vs in-flight.
ยง1Problem Statement
The lede above says what AgentCanvas is; this section is the why. Skip to ยง2 Capabilities for what it does.
Embodied agents โ spanning VLN (Vision-and-Language Navigation), EQA (Embodied Question Answering), and VLA (Vision-Language-Action) โ are increasingly built by composing foundation models with perception, mapping, memory, planning, and action. Unlike end-to-end policies, whose structure is absorbed into weights, this architecture is explicit and editable. That raises the question AgentCanvas is built around โ can agent design be searched rather than hand-built? โ alongside two stacks of pain it has to clear on the way.
Agent architecture is hand-built โ and could be searched
Each agent fixes a choice at every join โ sensor abstractions, map representations, memory state, prompt structure, planner topology, model placement, action interfaces โ by hand, usually for a single benchmark. As foundation models and embodied tools multiply, the space grows faster than manual iteration can cover, so the natural move is to search it rather than hand-tune it.
Agent Architecture Search (AAS) already does this for text-domain agents, but embodied transfer is not free: stateful simulators, noisy multi-episode scoring, long perception/action traces, and no off-the-shelf palette of embodied primitives. AgentCanvas is our attempt to supply the missing substrate โ a scaffold a coding agent can read, edit, run, and verify โ so that searching agent design becomes possible for embodied agents too.
Embodied-specific pain points
- Modern embodied stack is thick โ a working embodied agent needs LLM reasoning + tool use + simulator coupling + spatial tools, all wired together. Building this from scratch per project is prohibitively costly, and most of the effort goes into the execution layer rather than the idea being tested.
- Engineering nightmare โ an embodied agent is not one model but a
whole system โ a stateful simulator plus a stack of heavy models and tools. Just
running it, let alone at the scale benchmarking needs, is a hard engineering job in
itself:
- Python env hell โ no single Python env satisfies every part; each simulator, VLM, detector, and policy pins its own clashing CUDA / torch / Python, so finding one runtime they all share is often impossible โ you end up maintaining several incompatible environments just to load the agent.
- Batching โ each worker's simulator is a separate stateful process advancing at its own pace; you can batch the model but not the sims, so every step becomes an async gather-observations โ batch-infer โ scatter-actions dance.
- Other infra โ multimodal trajectories that must be logged and replayable, checkpoint/resume of multi-hour GPU runs that will crash, and debugging across process boundaries.
- Hidden ground-truth dependencies โ many methods quietly rely on simulator-provided ground truth (object poses, semantic labels, navigability) rather than real perception. Sometimes that's a legitimate way to control the experiment โ but, oversight or not, it often goes unmentioned in the paper.
Common AI research pain points (amplified here)
- Non-reproducible implementations โ every paper builds its agent
from scratch with a different codebase; comparing methods fairly or reproducing
results is painful โ and many of them are
Code coming SOON(Someday, Or Obviously Never). - Paper โ code โ papers show clean flow diagrams, but the actual code diverges in undocumented ways. Reproducing a paper means reverse-engineering its implementation.
- Tightly coupled code โ domain logic (prompts, tools, policies) is tangled with infrastructure. Swapping one component means rewriting the pipeline.
What the platform has to demonstrate as built & shipped
- 110+ node types โ 14 built-in plus ~100 domain types auto-discovered from
workspace/at startup; new types drop-in via Python files without framework edits. - Dataflow executor handling loops natively via IterIn/IterOut gates, with playback controls (play / pause / step / stop).
- Server-mode nodesets running in their own Python interpreters (Habitat 0.1.7 in
ac-vlnceenv, MatterSim inac-mp3d), auto-generated proxy nodes from declared port surfaces. - Real-time WebSocket streaming of observations / reasoning / actions / metrics, routed by
execution_idfor concurrent runs. - Multi-episode batch eval harness (
POST /api/eval/v2/startโ backend-ownedJobSchedulerspawns a subprocess-per-runBatchEvalRunner+Fleet) replacing the retiredvlnworkspace.evalHydra CLI. - Graph SDK โ the same graphs built and run as a Python library (
from agentcanvas import Graph), pip-installable from the repo-rootpyproject.toml, with catalog introspection, validation, run-event streaming, cassette record/replay, and graph โ code compilation. - Hot reload (
POST /api/components/reload) for new node types without server restart.
ยง2Capabilities
AgentCanvas is a visual agent design platform where researchers rapidly prototype embodied AI agents by drawing node graphs on a canvas. Graphs execute in real-time against simulator environments (Habitat, MatterSim/MP3D, SIMPLER, LIBERO) or real-world setups โ no pipeline code required.
Nine capabilities make this work โ the same nine as the capabilities section, one page each. Every card links to its page and opens a short implementation summary.
ยง2.1 Graph-Expressible Agents
The canvas supports workflow DAGs, loops (rollout or agent โ observe โ act โ repeat via IterIn/IterOut), and compositions of both. Researchers pick the pattern that fits their method. Capability page โ
How
TheGraphExecutor treats every graph uniformly โ nodes fire when inputs arrive. Cycles are handled by IterIn/IterOut gate nodes: when IterOut fires, it increments the step counter and transfers its outputs back to the paired IterIn, re-entering the loop. For pure DAGs, the same executor runs a single forward pass with no iteration. Both patterns coexist on one canvas โ a workflow DAG can contain an inner loop, or vice versa.
ยง2.2 Customizable Node System
New nodes are Python classes inheriting BaseCanvasNode or BaseNodeSet. Drop the file into workspace/, the platform auto-discovers it at runtime. No framework changes needed. Capability page โ
How
Authors subclassBaseCanvasNode and declare node_type, input_ports, output_ports, category, icon, and ui_config as class variables, then implement async forward(inputs, ctx) โ dict. The WorkspaceComponentRegistry scans workspace/ subdirectories at startup (and on hot-reload via POST /api/components/reload), imports every .py file, collects concrete BaseCanvasNode subclasses, and registers them in the node catalog (CATALOG). The frontend's GenericBlockRenderer renders any registered node automatically from its port and UI metadata โ no custom .tsx component needed. For groups of related tools, authors subclass BaseNodeSet instead โ a nodeset bundles multiple tools under shared initialize() / shutdown() lifecycle.
ยง2.3 Isolated Runtime Environments
Research tools often require conflicting Python environments (Habitat needs 3.8, SLAM needs ROS). Any BaseNodeSet can run in server mode in its own interpreter โ zero extra server code. Capability page โ
How
When a nodeset is loaded withmode="server", the WorkspaceComponentRegistry launches a subprocess using the nodeset's server_python interpreter. The subprocess runs AutoServerApp, which introspects the nodeset's tools โ reading their input_ports and output_ports โ and auto-generates an HTTP server with one endpoint per tool. The framework then fetches the server's manifest, generates proxy canvas nodes from it, and registers them in the node catalog. From the canvas perspective, proxy nodes behave identically to local nodes โ the researcher doesn't need to know whether a node runs in-process or in a remote subprocess.
ยง2.4 Nested Graph System
Any canvas graph can be saved as a graph node and dragged onto another canvas as a reusable block. Enables hierarchical agents โ a high-level planner containing sub-agent graph nodes. Capability page โ
How
AGraphDefinition can contain nodes whose .subgraph field holds another GraphDefinition. graphIn / graphOut boundary nodes define the composite's input/output interface. Before execution, flatten.py recursively expands all composites into a single flat graph โ prefixing inner node IDs for uniqueness and rewiring parent edges through the graphIn/graphOut boundaries. The executor only sees flat nodes. Graph nodes are stored as JSON in workspace/graph_nodes/ with snapshot semantics โ editing the source graph doesn't affect existing graph node instances.
ยง2.5 Graph Execution Engine
Nodes fire when inputs arrive, not in a fixed order. Supports cycles natively, enables real-time streaming, opens the door for future parallelism. Capability page โ
How
TheGraphExecutor maintains a ready queue. Entry nodes (no required inputs, no incoming edges) fire first. When a node fires, its outputs propagate along edges into downstream nodes' pending_inputs. A downstream node enters the ready queue once all its required ports have data. State is persistent across firings via _NodeStateProxy, and state containers (shared mutable state) are injected into nodes connected by state edges. A StopExecution exception from a termination node cleanly ends the loop.
ยง2.6 Execution Logs & Live Views
Every step streams observations, reasoning, actions, and metrics via WebSocket; every node firing is captured in structured per-run logs. Researchers see what the agent sees, read what it thinks, and can pause/step/resume โ then inspect any past run post-hoc. Capability page โ
How
Output viewer nodes (imageViewer, textScroll, actionLog, metrics, etc.) store their latest inputs. At each iteration boundary (when IterOut fires), _broadcast_step consolidates all output viewer data into a single nav_step WebSocket event โ including base64-encoded RGB/depth images, action names, LLM responses, state container snapshots, and metrics. The frontend renders these in real-time panels. An execution_id routes events to the correct client, enabling multiple concurrent agent runs with isolated output streams.
ยง2.7 Batch Evaluation & Job Queue
The same graph that runs on the canvas can be submitted as an eval job that scores it over hundreds of episodes. A backend-owned JobScheduler gates admission against a VRAM budget shared across all sessions; each admitted run is its own subprocess so backend restarts don't kill in-flight evals; per-episode logs and assets are self-contained on disk so a teammate can replay any single episode without re-running. Capability page โ
How
Two stacked layers (see capability page): Layer A isJobScheduler โ singleton with a per-second tick() that admits queued jobs when their calibrated per-resource estimates (VRAM + RAM) fit measured free minus standing reservations (declared marginal_vram_mb as fallback) and canvas Play isn't holding the exclusive lock; spawns the admitted job as python -m app.eval_subprocess_main --run-dir โฆ with PR_SET_PDEATHSIG so the kernel reaps orphans. Layer B is BatchEvalRunner in the subprocess โ reads spec.json/shared_urls.json, attaches to the parent backend's shared singletons via register_remote_nodeset (no duplicate VRAM), iterates the episode list through fresh GraphRun instances, pins each worker's chunk to its Fleet WorkerHandle when worker_count > 1, and writes self-contained episodes/ep{NNNN}/{log.jsonl, assets/, episode.json}. Shared singletons are refcounted across consuming jobs; the scheduler auto-unloads only those it loaded itself, so canvas-loaded copies stay (ADR-eval-001/002/003/004 + ADR-server-003).
ยง2.8 Visual Canvas Editor
The surface researchers actually touch: a React Flow editor where you drop nodes from a live backend catalog, wire typed ports, configure each node from an auto-rendered form, and watch every port update as the graph runs. The same component handles root canvas and composite drill-down โ no mode switching between design and observe. Capability page โ
How
Three layers stacked over one WebSocket (see capability page): ReactFlow (@xyflow/react) owns geometry; a tab-aware Zustand store (useFlowStore) owns graph semantics + per-node live nodeOutputs; a singleton WSManager with auto-reconnect carries nav_status (run state), nav_step (consolidated per-step bundle from graphOut sinks), viewer_data (per-sink-node {node_id, step, fields}), and error_event (ErrorBus envelopes). UnifiedGraphEditor reuses one component for mode="root" (live Zustand) and mode="subgraph" (local state + explicit save back to the parent composite's NodeDef.subgraph). PropertiesPanel auto-renders config forms from each node's ui_config.config_fields โ adding a new node type means adding its Python configSchema, not writing a panel. Typed-wire compatibility is checked client-side in useFlowStore.onConnect before the edge enters the store, so the canvas can never reach the backend in a state only validate_graph_connectivity would catch. The same store/WS also drives the Eval, Replay, and Log-viewer pages.
ยง2.9 Graph SDK
Build and run the same graphs as a Python library โ no canvas, no JSON authoring. from agentcanvas import Graph, add nodes, connect ports, run(): same nodes, same executor, same eval. Because it produces the same GraphDefinition, code โ JSON โ canvas stay fully reversible. Capability page โ
How
The SDK (app/graph_sdk.py, public namespace agentcanvas) is a thin ergonomic wrapper over the existing dataclass model (app/graph_def.py) and the in-process GraphExecutor โ it adds no execution machinery. catalog() / describe() / nodesets() introspect the live node catalog; generate_node_stubs() emits typed authoring stubs so a coding agent gets IDE-checkable port names; validation raises GraphValidationError before anything runs; run() returns a RunResult with an optional on_event RunEvent stream, and eval() drives batch evaluation. Cassette record/replay (graph_sdk_replay.py) captures server-node HTTP round-trips so GPU-nodeset graphs (Habitat, SAM) replay as hermetic tests; graph_to_code() (graph_sdk_codegen.py) compiles any existing graph back into a builder script. Distributed as pip install agentcanvas from the repo-root pyproject.toml โ the core surface is stdlib-light, with tiered extras [server] / [llm] / [backend].
ยง3Design Principles
-
Framework has zero domain knowledge
agentcanvas/backend/app/is a pure execution engine. All domain-specific code (policies, prompts, tools, environments) lives inworkspace/. The framework never imports domain code directly โ it discovers and bridges components at runtime. - Graph, not script Agent behavior is defined as a dataflow graph, not imperative code. This makes architectures visual, composable, and inspectable. The graph is the source of truth โ not a visualization of code, but the code itself.
- Dataflow execution Nodes fire when their inputs arrive, not in a fixed order. Naturally supports loops (observe โ act โ repeat) and enables future parallelism and branching.
- Composability via nesting Composites let researchers group nodes into reusable sub-graphs at any depth. Execution always flattens to a single dataflow graph โ no outer/inner split at runtime. Keeps the executor simple while enabling arbitrarily deep hierarchical agent architectures.
- Real-time observability Every step streams observations, reasoning, actions, and metrics via WebSocket. Researchers see what the agent sees, read what it thinks, and can pause/step/resume at any point.
-
Extend, don't fork
New capabilities are added by dropping Python files into
workspace/, not by modifying framework code. The component system auto-discovers new tools, skills, agents, environments, policies, and node handlers.
ยง4Core Components
| Component | Location | Role |
|---|---|---|
| Canvas Editor | agentcanvas/frontend/ |
Visual graph editor (React Flow + Zustand) |
| Graph Executor | agentcanvas/backend/app/agent_loop/graph_executor.py |
Single data-driven executor โ fires nodes on input arrival; handles both DAG workflows and cyclic loops via IterIn/IterOut gates (no separate DAG executor) |
| Loop Runner | agentcanvas/backend/app/agent_loop/loop_runner.py |
Singleton that manages pause/stop/resume lifecycle |
| Component Registry | agentcanvas/backend/app/components/registry.py |
Auto-discovers workspace/ components, bridges into runtime |
| Wire Type System | agentcanvas/backend/app/standard/wire_types.py |
9 current typed edges (IMAGE, DEPTH, POSE, DISCRETE_ACTION, CONTROL, TEXT, BOOL, METRICS, ANY) + LIST[T] modifier; 3 deprecated names (ACTION, OBSERVATION, STEP_RESULT) stay registered so legacy graphs load |
| Node Handlers | agentcanvas/backend/app/agent_loop/builtin_nodes.py + workspace/nodesets/ |
14 built-in node types + nodeset-provided domain nodes, all registered in the node catalog (CATALOG) |
| Env Nodesets | workspace/nodesets/env/ |
Gym-like env nodesets (env_habitat, env_mp3d, env_simpler, env_libero, env_hmeqa, โฆ); the framework-level BaseEnvManager abstraction was removed (ADR-components-002) |
| Eval Harness (WIP) | app/services/job_scheduler.py + agent_loop/eval_batch.py (BatchEvalRunner) + agent_loop/graph_run.py + components/fleet.py + POST /api/eval/v2 + frontend Eval page |
Subprocess-per-run JobScheduler with cross-session measured resource admission (VRAM + RAM), worker-pool + batched-inference (ADR-eval-002/003/004). Replaces the retired vlnworkspace.eval Hydra CLI โ Eval UI is still being built (Feature TODO F5). |
| Graph SDK | agentcanvas/backend/app/graph_sdk.py (+ _replay, _codegen; public namespace agentcanvas) |
Programmatic authoring + in-process running of the same GraphDefinition โ builder, catalog, validation, run events, record/replay, graph โ code |
| User Workspace | workspace/ |
All domain components: nodesets (methods, envs, models), graphs, graph nodes |
ยง5Core Features
Visual agent composition
- Unified canvas paradigm (ADR-canvas-002): one editor, one node registry (110+ types), one catalog at every graph level โ root canvas and subgraphs use identical tools
- Flat workspace (ADR-canvas-001): all node types on one canvas โ no outer/inner layer split
- Drag-drop node editor with optional recursive composite nodes (ADR-dataflow-001)
- Pre-built graph templates (Straightforward CMA, NavGPT-CE) โ loaded as flat workspace JSON
- Type-checked wiring with 9 wire types and color-coded handles
- Unified sidebar catalog: Environment, LLM, Prompt, Loop Control, Output, Composite, Agent Presets, Saved Graphs
- Graph save/load: two kinds โ Graphs (editable templates in
workspace/graphs/, open in tabs) and Graph Nodes (frozen composites inworkspace/graph_nodes/, drag onto canvas as reusable blocks with snapshot semantics). Explorer sidebar shows both as separate sections with VS Code-style tree expansion (ADR-canvas-003) - Composite nodes: optionally group nodes into reusable composites with graphIn/graphOut boundaries
- Execution toolbar: Play/Pause/Stop/Step/Reset controls at page level (no AgentLoop wrapper needed)
NodeSet manager
- Dedicated page for managing NodeSets, server-mode nodesets, and environments โ separate from canvas
- Load/unload NodeSets: activation makes tools appear in canvas sidebar, deactivation removes them
- Start/stop/restart server-mode nodesets (Habitat, MatterSim, SIMPLER, etc.)
- Rescan
workspace/for new components - Auto-hosted deployment (ADR-server-001): any BaseNodeSet can be loaded in server mode โ framework auto-generates the server from tool port definitions, zero extra code
Real-time execution & streaming
- Single
/runendpoint: send graph definition + execution_id, get WebSocket events - 8 WebSocket event types:
nav_step,nav_llm_step,nav_status,nav_complete, and more - Playback controls: play, pause, resume, step, stop
execution_idrouting: multiple concurrent agent runs with isolated output streams
Iteration & cycles
- Two-sided IterIn/IterOut gates for clean cyclic graphs (no back-edges) โ IterIn captures run-start seeds on its left side; IterOut carries loop state and a
stopinput that halts the loop - Configurable step budget (default 500)
- Loop ends when the IterOut
stopport goes true, the budget is exhausted, or the user stops โ the verdict stage runs once on the after-loop band (the former Termination node was removed; see glossary ยง6)
Extensible component system
- 2 base classes:
BaseCanvasNode,BaseNodeSet - Auto-discovery: drop a Python file in
workspace/, reload, it's available - Lifecycle hooks: async
initialize()/shutdown() - Hot reload:
POST /api/components/reload - Saved composite graphs via
/api/graphs(CRUD forworkspace/graphs/*.json) - Hook system (
agent_loop/hooks.py): shell commands fire at graph/node boundaries โ 5 events (GraphStart/GraphComplete/GraphError/PreNodeExecute/PostNodeExecute), JSON event payload on stdin, JSON{"action": "continue"|"block"|"modify"}response on stdout, fail-open: any hook error (timeout, crash, malformed JSON) continues execution unchanged
State container system (ADR-dataflow-002)
- Dual-wire architecture: data edges (typed dataflow) + state edges (access grants to shared state containers) coexist on canvas
- StateContainer: dict of named
BaseStateentries โ each with a reducer (accumulator, lastWrite, counter), a value type from an extensible registry, and alifetimefield (forever / step / episode / run / custom) that subscribes the state to framework signals for declarative clearing - State value types: extensible registry โ TEXT, BOOL, METRICS, POSE plus domain types (POINTCLOUD, OCCUPANCY_MAP, EMBEDDING, EPISODE_CONTEXT) and ANY
- State edges: separate wire system โ dashed violet lines granting read/write access, toggled via toolbar button,
__state__port on every node - graph_state: optional graph-level container auto-accessible by all nodes โ displayed in State tab of bottom panel
- Node decomposition: "fancy" method nodes (e.g. a history tracker or a reason step) can be decomposed into composite graphs with visible internal state
- Checkpointable: each state implements checkpoint/restore for persistence and replay
VLN-CE evaluation
- Native path: Habitat policies with
.act()interface - External path: custom models with
.predict_action()interface - Metrics: SPL (primary), SR, nDTW, SDTW + per-episode breakdown
ยง6Differentiation
AgentCanvas combines three structural features that are collectively absent from current agent frameworks:
| Framework | Externalised topology | Typed ports | Nodeset boundaries | All three? |
|---|---|---|---|---|
| AgentCanvas | โ Graph JSON | โ IMAGE / DEPTH / DISCRETE_ACTION / CONTROL / POSE / TEXT / METRICS / โฆ | โ Loadable nodesets with declared tool surface | โ |
| LangGraph | ~ Topology built imperatively via Python add_node / add_edge; not externalised as a single artefact |
โ Edges carry arbitrary Python state objects | ~ Tool registries exist but no declared-equivalence semantics | โ |
| dspy | โ Compute graph is implicit in the optimised Module program | ~ Signatures type the I/O of one module, not the inter-module wires | โ No nodeset analogue | โ |
| PocketFlow | โ Flow JSON exists | โ Ports are untyped (any Python object) | โ No nodeset analogue | โ |
| Ad-hoc Python harness (typical paper repo) | โ Topology = the source code | โ Untyped function signatures | โ Tools are imports | โ |
The triple matters because it makes the platform's content (graphs) structurally inspectable: graph topology is a JSON artefact, port types are declarable and checkable, and tool boundaries are explicit nodeset surfaces. This is what makes AgentCanvas a viable substrate for downstream automation โ for example, an automated faithfulness audit can reduce its rules to JSON-checkable structural questions only because of these surfaces. Without all three, audit cost dominates run cost.
- Not that other frameworks are inferior for every use case โ LangGraph offers stronger imperative flexibility, dspy stronger compiler-level optimisation. AgentCanvas trades imperative flexibility for declarative auditability.
- Not that AgentCanvas is feature-complete vs the LangChain ecosystem โ LangChain has hundreds of pre-built integrations; AgentCanvas has ~110 node types in v1.
- Not that the triple is uniquely sufficient for any agent design task โ only necessary for tasks that benefit from structural inspectability (auditing, search, transfer benchmarking).
ยง7Open Questions
- Streaming graph mutations โ runtime in-graph mutations (add/remove/rewire nodes mid-execution) vs current snapshot-and-reload model. Latter is simpler and what the AAS architecture-search loop currently uses; former unlocks self-modifying agents but multiplies executor complexity.
- State container reducer DSL โ Python callable per reducer (current) vs declarative DSL (e.g. JSON spec โ reducer factory). Latter would let reducers ride in graph JSON for full self-containment; former is more flexible.
- Multi-tenant execution โ single global executor (current; single-user assumption) vs per-user executor pool with isolation. Becomes load-bearing only if AgentCanvas ships as a hosted service.
- Graph node snapshot semantics โ copy-on-save (current; immutable graph nodes) vs reference (live updates propagate from source graph). Current is safer; reference would feel more git-like and reduce stale-snapshot drift.
- graphIn/graphOut arity โ fixed 1-port boundary (current) vs N-port boundary (multiple inputs/outputs per composite). Fixed simplifies the flatten algorithm; N-port matches researcher mental model better.
ยง8Status
shipped (main platform) ยท drafting (Eval harness UI โ Feature TODO F5)
Core platform stable: graph executor, IterIn/IterOut iteration, 110+ node
types, server-mode nodesets (Habitat / SIMPLER / Prismatic), WebSocket
streaming, NodeSet Manager, hook system, Graph SDK (programmatic authoring +
pip install agentcanvas). Bottleneck: Eval
harness frontend (Feature TODO F5) is incomplete โ backend works but the Eval
page in agentcanvas/frontend/src/eval/ still has gaps. Runtime
graph mutation โ letting the AAS search loop edit graphs through typed
operations rather than ad-hoc file rewrites โ is the headline
v2 (Topology-Growing Canvas) capability,
out of scope for the v1 bounded-topology paradigm.
Next: finish F5 frontend.