AgentCanvas

Embodied agents have no shared substrate

Every embodied agent is bespoke plumbing — perception, memory, planning, and action hand-wired in Python and bolted to a single simulator. Swapping a module, swapping the simulator, or running the same agent across navigation, QA, and manipulation means rewriting the glue each time; heavyweight simulators reload per worker; orchestration is a hand-coded loop; and results rarely reduce to a number someone else can reproduce.

AgentCanvas makes the agent a single typed graph — one JSON is one agent — run by a graph executor where typed wires and ports drive scheduling, simulators live as shared server-mode services, and nodes auto-discover with no central registry. The agent becomes a program: configurable, swappable, run at scale, and reproducible to a fixed SR / SPL / accuracy on one substrate that spans all three domains.

A typed graph, run by an executor

An executor is one node-and-wire JSON graph — the complete topology, with no hidden Python state; whatever mutates lives in state containers, not globals. Each node is a Python module that exposes typed input / output ports, and nodes are auto-discovered from nodeset libraries: env (simulators), method (the agent logic), model (LLM / VLM services), and policy (neural controllers) — swap a nodeset to swap a capability.

A graph executor runs it: typed wires connect compatible ports, and those wires drive the scheduling — there is no hand-coded orchestration loop, and a loop is just a back-edge.

Each nodeset, its own interpreter

Embodied research tools demand mutually incompatible Python environments — Habitat-Sim pins Python 3.8, modern SAM / HF models need torch ≥ 2, ROS-based SLAM wants its own stack. They cannot share one interpreter. So any nodeset can run in server mode: its own subprocess, its own interpreter (a named conda env), GPU-isolated.

There is no server code to write. AgentCanvas introspects the nodeset's typed ports and auto-generates an HTTP plugin server (/manifest + /call), then drops proxy nodes on the canvas that look and wire up exactly like local ones — an author never knows, or cares, which interpreter a node lives in. The same server mode is what fans a simulator nodeset out to N isolated workers under batch eval.

A server-mode node pays an IPC hop per call — typed payloads cross the process boundary as serialized values — so lightweight inner-loop logic stays local, while heavyweight or dependency-conflicting tools (simulators, big models) run server-side.

Autonomous evaluation, batched at scale

Pick a graph and a worker count, then Start: AgentCanvas evaluates candidates across the simulator suite with no human in the loop. The same server mode that isolates a nodeset now fans it out — N tagged copies of the env subprocess (#0, #1, …) step concurrently, draining one shared queue of episodes, while every shared foundation-model node stays a single batched service the workers call together.

So weights load once, not per worker — VRAM stays flat — and there is no lockstep barrier: a naive batch eval makes every worker wait on the slowest module each step, but here workers proceed independently and the model batches whoever is ready within a short window. Throughput is bound by per-worker speed, not the slowest module; at worker count 1 the run is bit-identical to a plain sequential eval.

Typed logs that render themselves

AgentCanvas logs every node firing — its typed input and output ports, tagged with step and firing index — alongside the internal breadcrumbs a node chooses to write: the planner's thought, its prompt, the parsed action, each tool call. So a candidate edit connects to the exact firings and decisions it caused.

Because the ports are typed, the log renders itself: RGB, depth, and the 36-view panorama come back as inline thumbnails, actions and metrics as structured chips and tables — not opaque blobs. Heavy values are offloaded to sidecar assets and referenced by path, so the record stays light.

Access to the evidence is necessary but not sufficient: the logs are there, but an optimizer still has to decide which to inspect and how to turn it into the next edit — credit assignment does not come for free.

logs.detailrecorded live · 36-view panorama

What you can do with it

From building one agent to letting a coding agent search thousands — the same typed-graph substrate underneath.

Build, run, and search agent architectures

Build and run embodied agents as typed graphs, evaluate them in simulation at scale, and — when you want it — let a coding agent search architectures on the same substrate. The code, developer guide, tutorials, and NodeSet references are all online; AgentCanvas is also the runtime behind the paper Automating the Design of Embodied Agent Architectures.

Citation

If AgentCanvas or its embodied Agent Architecture Search results are useful in your research, please cite the paper.

@misc{jian2026AgentCanvas,
  title         = {Automating the Design of Embodied Agent Architectures},
  author        = {Jian Zhou and Sihao Lin and Jin Li and Shuai Fu and Gengze Zhou and Qi Wu},
  year          = {2026},
  eprint        = {2606.30111},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2606.30111}
}