AgentCanvas
One typed graph, two roles — a harness that runs embodied agents, and a scaffold that lets coding agents edit and verify them. Perception, memory, planning, and action become node-and-wire programs that execute live inside stateful simulators, with typed ports, batch-optimized evaluation, and episode-level logs that serve both.
Embodied agents have no shared substrate
Every embodied agent is bespoke plumbing — perception, memory, planning, and action hand-wired in Python and bolted to a single simulator. Swapping a module, swapping the simulator, or running the same agent across navigation, QA, and manipulation means rewriting the glue each time; heavyweight simulators reload per worker; orchestration is a hand-coded loop; and results rarely reduce to a number someone else can reproduce.
AgentCanvas makes the agent a single typed graph — one JSON is one agent — run by a graph executor where typed wires and ports drive scheduling, simulators live as shared server-mode services, and nodes auto-discover with no central registry. The agent becomes a program: configurable, swappable, run at scale, and reproducible to a fixed SR / SPL / accuracy on one substrate that spans all three domains.
A typed graph, run by an executor
An executor is one node-and-wire JSON graph — the complete topology, with no hidden Python state; whatever mutates lives in state containers, not globals. Each node is a Python module that exposes typed input / output ports, and nodes are auto-discovered from nodeset libraries: env (simulators), method (the agent logic), model (LLM / VLM services), and policy (neural controllers) — swap a nodeset to swap a capability.
A graph executor runs it: typed wires connect compatible ports, and those wires drive the scheduling — there is no hand-coded orchestration loop, and a loop is just a back-edge.
Each nodeset, its own interpreter
Embodied research tools demand mutually incompatible Python environments — Habitat-Sim pins Python 3.8, modern SAM / HF models need torch ≥ 2, ROS-based SLAM wants its own stack. They cannot share one interpreter. So any nodeset can run in server mode: its own subprocess, its own interpreter (a named conda env), GPU-isolated.
There is no server code to write. AgentCanvas introspects the nodeset's
typed ports and auto-generates an HTTP plugin server
(/manifest + /call), then drops proxy nodes on the canvas that
look and wire up exactly like local ones — an author never knows, or cares, which interpreter a node
lives in. The same server mode is what fans a simulator nodeset out to N isolated workers under batch
eval.
A server-mode node pays an IPC hop per call — typed payloads cross the process boundary as serialized values — so lightweight inner-loop logic stays local, while heavyweight or dependency-conflicting tools (simulators, big models) run server-side.
Autonomous evaluation, batched at scale
Pick a graph and a worker count, then Start: AgentCanvas evaluates candidates across the
simulator suite with no human in the loop. The same server mode that isolates a
nodeset now fans it out — N tagged copies of the env subprocess
(#0, #1, …) step concurrently, draining one shared queue of episodes, while
every shared foundation-model node stays a single batched service
the workers call together.
So weights load once, not per worker — VRAM stays flat — and there is no lockstep barrier: a naive batch eval makes every worker wait on the slowest module each step, but here workers proceed independently and the model batches whoever is ready within a short window. Throughput is bound by per-worker speed, not the slowest module; at worker count 1 the run is bit-identical to a plain sequential eval.
Typed logs that render themselves
AgentCanvas logs every node firing — its typed input and output ports, tagged with step and firing index — alongside the internal breadcrumbs a node chooses to write: the planner's thought, its prompt, the parsed action, each tool call. So a candidate edit connects to the exact firings and decisions it caused.
Because the ports are typed, the log renders itself: RGB, depth, and the 36-view panorama come back as inline thumbnails, actions and metrics as structured chips and tables — not opaque blobs. Heavy values are offloaded to sidecar assets and referenced by path, so the record stays light.
Access to the evidence is necessary but not sufficient: the logs are there, but an optimizer still has to decide which to inspect and how to turn it into the next edit — credit assignment does not come for free.
What you can do with it
From building one agent to letting a coding agent search thousands — the same typed-graph substrate underneath.
Drag nodes from the nodesets, wire their typed ports, press Play, and watch it run live in the simulator.
Re-implement MapGPT, NavGPT, Explore-EQA, or VoxPoser as a typed graph and hit its reported numbers.
Swap only the env nodeset to take the same agent from navigation (R2R) to EQA (HM-EQA) to manipulation (LIBERO).
Fan a candidate out across episodes and workers, then read SR / SPL / success on a public split.
Scroll the typed episode log — the panorama, the planner's thought, the action — firing by firing, to see why it failed.
Drive builds and evals over HTTP / MCP so a coding-agent optimizer proposes edits and verifies them in simulation.
Build, run, and search agent architectures
Build and run embodied agents as typed graphs, evaluate them in simulation at scale, and — when you want it — let a coding agent search architectures on the same substrate. The code, developer guide, tutorials, and NodeSet references are all online; AgentCanvas is also the runtime behind the paper Automating the Design of Embodied Agent Architectures.