Architecture — AgentCanvas

§1System Overview

Three-tier system: frontend (React canvas) ↔ backend (FastAPI + GraphExecutor) ↔ workspace (user-authored Python components + simulators).

§2Data Flow

A single execution cycle (ADR-canvas-002: unified canvas):

User clicks Play in the ExecutionToolbar
Frontend generates execution_id (UUID)
getGraphForExecution() serializes the root canvas as GraphDefinition (filters out the frontend-only stateContainer node type — containers travel as graph.containers; output viewers are backend nodes and are sent along)
POST /api/navigate/run { loop_definition, execution_id }
Backend creates GraphExecutor from loop_definition
Nodes fire by dataflow as inputs arrive: env observe → LLM call → parse → env step (domain nodes are nodeset-namespaced <nodeset>__<verb> types; built-ins like llmCall / textParse fill the reasoning slots)
IterOut collects end-of-step data → IterIn provides it next iteration
Each node emits WebSocket events tagged with execution_id
Frontend discovers output nodes by type, routes WS events to them
Loop ends: IterOut's stop port goes true, the step budget is exhausted, or user Stop — the verdict stage then runs once on the after-loop band
nav_complete event carries final metrics (SPL, SR, nDTW, SDTW)

§3Flat Workspace (ADR-canvas-001)

The canvas is a ComfyUI-style flat workspace. All node types coexist on one surface — environment, model, reasoning, control, and output nodes are wired freely without layer boundaries.

Category	Nodes	Purpose
Environment	`<env>__reset` / `__step` / `__observe` / `__metric`, …	Interact with simulator via gym-like verbs (from activated env NodeSets)
Model	`llmCall` (built-in) + NodeSet model/policy nodes (e.g. `policy_adapter_vlnce__predict`, SAM)	Run neural policies or language models
Reasoning	`textParse`, `historyLog` (built-ins) + method NodeSets' reason/parse nodes	Build prompts, make decisions
Control	`iterIn`, `iterOut`	Manage iteration cycles; iterOut's `stop` port and after-loop band handle loop exit + verdict
Output	`imageViewer`, `textViewer`, `textScroll`, `actionLog`, `metrics`	Display streaming results
Tool	(dynamic from NodeSets)	SAM, navigation tools, etc.

The workspace graph is a GraphDefinition — save/load the entire canvas as JSON via /api/graphs.

3.1 Unified Canvas Paradigm (ADR-canvas-002)

One editor, one node registry, one catalog at every graph level:

unifiedNodeTypes.ts: 3 explicit entries (compositeNode, stateContainer, _generic) + a Proxy routing every other registered type to GenericBlockRenderer; the full catalog enumeration lives in unifiedCatalog.ts
unifiedCatalog.ts: single sidebar catalog with unified categories
UnifiedGraphEditor.tsx: dual-mode editor (root: Zustand store, subgraph: local state with save-back)
ExecutionToolbar.tsx: standalone Play/Pause/Stop controls (root canvas graph sent directly to /run)
Graph save/load: sidebar shows saved graphs from /api/graphs, draggable as composites, loadable as root
Composites (ADR-dataflow-001) preserved for optional nesting — same editor inside

3.2 NodeSet Manager

NodeSets are managed on a separate page (not the canvas). When a NodeSet is activated, its tools appear in the canvas sidebar. When deactivated, they disappear.

3.3 Recursive Composite Nodes (ADR-dataflow-001)

Composites remain available for optional grouping — any group of nodes can become a reusable composite with defined I/O, nestable to arbitrary depth.

Key concepts:

CompositeNode: a node that contains a GraphDefinition subgraph
graphIn / graphOut: boundary nodes defining composite inputs/outputs (separate from IterIn/IterOut)
Canvas Stack: navigation breadcrumbs appear when inside a composite
Flatten-before-execute: backend recursively expands composites into a flat graph before execution

Author — select nodes → Group → a composite with auto-generated graphIn/graphOut boundaries.
Save — written to workspace/graphs/*.json → appears in the sidebar catalog, draggable onto any canvas.
Execute — flatten_graph() expands every composite → the GraphExecutor runs one flat graph.

§4Execution Engine

One engine — the Graph Executor — handles every graph regardless of topology. DAG-shaped workflows are just cyclic graphs that fire each node exactly once; iterative agent loops are the same engine with iterIn/iterOut gates added. There is no separate "DAG executor" path (see Glossary §6).

4.1 Graph Executor

Nodes fire when all required inputs arrive — data-driven, not topo-sort-driven
Two-sided iteration model (ADR-dataflow-008, folding ADR-dataflow-006's three pivots to two): iterIn is two-sided — its left input ports (declared in initPorts) capture run-start seeds once into init_ slots, its right output ports broadcast step boundaries + expose the loop-carry bundle; iterOut collects end-of-iter state and transfers it back to iterIn via pairedWith for the next iteration. The former Initialize pivot is folded into iterIn's left side; the initialize node type was removed entirely on 2026-06-10 — graphs still carrying one are rejected at validation with a migration hint.
Per-node state via _NodeStateProxy (tracks inputs, outputs, fire count)
Supports pause/resume/step via LoopRunner
Two-sided iterOut (ADR-dataflow-009): the former Termination node folded into iterOut's stop input port (checked once per iteration by the Decide phase); on termination iterOut emits a final side (final_<name> + final_stop) exactly once, feeding the after-loop band (evaluate → graphOut) that holds the terminal iteration's values
Multi-scope (ADR-dataflow-007): N coexisting (iterIn, iterOut) pairs in one flat graph, each with its own iteration cadence; scope forest computed by analyze_scopes()
Flatten step: at the top of run(), flatten_graph() recursively expands any composite nodes before processing
Startup seed-discovery rule: a node is queued at run-start iff its type is not iterIn AND it has no incoming edges AND it has no required input ports. Required-but-unwired ports are rejected at the API boundary by validate_graph_connectivity() in graph_def.py — no silent never-fires.

4.2 Headless Eval Path (JobScheduler)

The same engine, driven without the canvas — the platform's primary path for producing numbers. §2's interactive Play runs one episode; this runs hundreds.

POST /api/eval/v2/start enqueues a job on the backend-owned JobScheduler (services/job_scheduler.py)
A per-second tick() admits queued jobs when their calibrated per-resource estimates (VRAM + RAM) fit measured free minus reservations (declared marginal_vram_mb as fallback) and canvas Play isn't holding the ExecutionGuard lock (state.py — canvas/eval mutual exclusion)
Each admitted run is its own subprocess — python -m app.eval_subprocess_main --run-dir …, kernel-tied to the backend by PR_SET_PDEATHSIG. It re-scans workspace/ from disk (fresh code per launch) and attaches to parent-owned shared singletons via register_remote_nodeset — no duplicate VRAM
Inside the subprocess, BatchEvalRunner (agent_loop/eval_batch.py) iterates the episode list through fresh LoopRunner instances; at worker_count>1 an EnvWorkerPool leases tagged env subprocesses per episode
Results land self-contained under outputs/eval_runs/{run_id}/episodes/ep{NNNN}/ (log.jsonl, assets, episode.json); spec.json / shared_urls.json / _DONE form the parent↔subprocess file protocol (services/run_state_io.py)

Governed by ADR-eval-001…004 + ADR-server-003 — see the Batch Eval design-doc.

§5Component System

Scan order: nodesets/ → nodes/; graphs/ + graph_nodes/ are pure data. Hot reload re-runs the scan via POST /api/components/reload.

5.1 Auto-Hosted Deployment (ADR-server-001)

Any BaseNodeSet can be loaded in server mode without writing a ServerApp. The registry launches auto_host in a subprocess, which uses AutoServerApp to introspect the nodeset's tools and auto-generate server functions from their port definitions. The result is identical to YAML-based servers — proxy nodes are registered via generate_proxy_nodes().

POST /api/components/nodesets/sam/load?mode=server
spawns python -m app.server.auto_host --file …/sam.py --class SamNodeSet --port 9200
fetches /manifest, generates proxy nodes via generate_proxy_nodes()
canvas sees identical nodes to YAML-declared servers

Endpoint / path	Role
`workspace/graphs/`	Saved graph templates (`kind="graph"`)
`workspace/graph_nodes/`	Archived composite nodes (`kind="node"`)
`GET /api/graphs`	List all (both dirs)
`POST /api/graphs`	Save — routes by `kind` field
`DELETE /api/graphs/{id}`	Delete (searches both dirs)

5.2 Python-Driven Node UI (ADR-components-004)

Every BaseCanvasNode carries a ui_config ClassVar (NodeUIConfig) that declares how the node renders on the canvas — color, layout, dimensions, and inline config controls. The frontend's GenericBlockRenderer reads ui_config at render time, eliminating the need for per-node .tsx components.

Two layouts: Block (default — standard rectangle with ports, label, optional config fields) and Strip (narrow vertical gate, used by iterIn, iterOut, graphIn, graphOut).

ConfigField declares inline widgets on the node face: label, slider, text, select, toggle, textarea. Example: LLMCall shows a temperature slider; Decision shows an action-space dropdown.

This reduced unifiedNodeTypes.ts to 3 explicit entries — compositeNode, stateContainer, and a _generic Proxy fallback that routes every other node type (built-ins + NodeSet tools) to GenericBlockRenderer. The former per-viewer custom .tsx components are gone: viewers now render as display-field layouts inside GenericBlockRenderer (ImageViewerField, LogListField, MetricTableField, TextViewerField under nodes/agentloop/inner/layouts/fields/). The remaining custom components handle interactions that exceed port+config rendering:

Custom `.tsx`	Why
`CompositeCanvasNode` / `CompositeNodeView`	Subgraph preview, port derivation, drill-in
`StateContainerNode`	Canvas-visible state container element (dual-wire system §6) — frontend-only type, filtered from execution

The /node-schemas API endpoint includes ui_config in its response, so the frontend catalog can enrich sidebar entries with backend-defined styling at drag time.

2 base classes define the extension points:

Base Class	What It Provides
`BaseCanvasNode`	Custom graph node handler (forward method, `NodeUIConfig` for Python-driven UI, typed port declarations) — see ADR-components-004
`BaseNodeSet`	Atomic group of tools (load/unload together, optional server mode)

5.3 Runtime Services Around the Engine

LLM provider layer (app/llm/) — llmCall resolves its profile through ProfileStore → provider registry (24 providers via litellm) with a layered env-var fallback (ADR-platform-003); llm_complete() / vlm_complete() are the call seam
Env panels (components/env_panel.py + /api/env-panels + frontend EnvPanel.tsx) — per-nodeset control plane for env-side runtime knobs (suite / split / episode, play / pause / reset); renamed from "controllers" 2026-06-11
File-watch hot reload (services/graph_watcher.py, services/nodeset_watcher.py) — graph JSON and nodeset .py edits made on disk (e.g. by a coding agent) propagate without a restart, beyond the manual POST /api/components/reload
Episode replay (app/replay/ + /api/replay + frontend Replay page) — env nodesets opt in via a replay_parser that turns persisted episode logs into an env-agnostic timeline
Monitoring (services/resource_sampler.py + services/system_runs.py + frontend Monitor page) — always-on ~1 Hz resource time-series plus per-run performance summaries
Structured per-run logs (app/logging/logger.py) — automatic exterior port-I/O capture + voluntary _self_log() interior detail, persisted as JSONL (ADR-observability-003)

§6State Container System (ADR-dataflow-002)

A dual-wire architecture adds visible shared state alongside the existing dataflow edges:

6.1 Container Model

A StateContainer is a dict of named BaseState entries. Each state has two axes:

Reducer type: Accumulator (list.append), LastWrite (overwrite), Counter (sum) — the ephemeral reducer was removed 2026-04-18
Value type: narrow set for memory (TEXT, BOOL, METRICS, POSE, POINTCLOUD, OCCUPANCY_MAP, EMBEDDING, EPISODE_CONTEXT, ANY — raw wire payloads rejected, ADR-dataflow-004)
Lifetime: forever / step / episode / run / custom — expands to a reset_on signal list; per-iteration clearing is lifetime="step" (replaces the old Ephemeral reducer)

6.2 Two Wire Systems

	Data Edges	State Edges
Carry	Typed data per-firing	Access grants (no data)
Visual	Solid animated lines	Dashed violet, no arrowhead
Created via	Port handles (left/right)	`__state__` handle (bottom)
Visibility	Always	Toggle on/off

6.3 graph_state

Optional graph-level container auto-accessible by all nodes — no state edges needed. Displayed in the State tab of the bottom OutputDrawer panel. Each composite can have its own scoped graph_state.

6.4 Executor Integration

Containers built after flatten_graph(), before seed-node discovery
Connected containers injected into ctx._containers before each node fires
graph_state injected into ctx._graph_state for all nodes
broadcast_signal("step_end", {"step": N}) fires at IterOut — states with lifetime="step" clear via their reset_on subscription
Container snapshots broadcast in nav_step WebSocket event

See State Containers (design-doc) for the full design.

§7Tech Stack

Layer	Choice	Why
Backend runtime	Python 3.10+ (agentcanvas env), FastAPI	Modern ML stack + async web framework (ADR-platform-004)
ML framework	PyTorch 2.5 + CUDA 12.1 (agentcanvas)	Modern torch for SAM, HF, LLM nodesets
Simulator	Habitat-Sim 0.1.7 (`ac-vlnce` env, Python 3.8)	Binary locked to 3.8; accessed via server mode (ADR-server-001)
Frontend framework	React 18 + TypeScript	Component model, strict typing
Canvas library	`@xyflow/react` (React Flow)	Mature node-graph editor with customization
State management	Zustand (dual store)	Lightweight, no boilerplate, works with React Flow
Styling	Tailwind CSS	Utility-first, fast iteration
Build tool	Vite	Fast HMR for frontend dev
Config (eval)	Hydra + OmegaConf	Composable config groups for eval harness
Docs	Hand-authored HTML + stdlib dev server (`docs/run_dev.sh`)	Full HTML expressivity; live reload via SSE on mtime change. MkDocs retired 2026-05-18.