Installation

AgentCanvas is not one environment — it is one orchestrator hub (the agentcanvas conda env) surrounded by a fleet of satellite envs, one per simulator or model. Part 1 is the mental model: what envs exist and how they relate. Part 2 is the mechanics: how to install each one and the flow they share.

Part 1. Mental Model

Everything runs through the agentcanvas env: it hosts the backend, the frontend, the graph executor, every method / reasoning nodeset (NavGPT, MapGPT, AO-Planner, …), and all LLM calls. It deliberately never imports a simulator. Each heavy dependency that can't coexist with that stack — Habitat-Sim 0.1.7 (pinned to Python 3.8), MatterSim (its own C++ build), the VLA policy stacks (their own torch/JAX), DetAny3D (CUDA 11.8) — lives in its own satellite env and is reached over server mode (the backend auto-hosts it as a subprocess and talks to it over HTTP). So the picture is hub-and-spoke:

The same picture as a list. One hub + eleven satellite envs. Two installers create no env at all: fetch_ckpt_rt1.sh just downloads RT-1-X checkpoints (used from the VLA envs) and fetch_dataset_openeqa.sh just stages the OpenEQA dataset (the EM-EQA reasoner runs inside agentcanvas with gpt-4o). agentcanvas-test is a throwaway clone of the hub for CI.

Why so many. Two forces: (1) the framework import boundary — the backend in agentcanvas/backend/app/ may never import habitat, habitat_sim, vlnce_baselines, or habitat_baselines; and (2) mutually incompatible pins — Habitat-Sim 0.1.7 needs Python 3.8, DetAny3D needs CUDA 11.8, the VLA stacks need their own torch/JAX, and the frozen FM research stacks (Prismatic, RAM, SpatialBot) pin old transformers that the shared ac-fm env's 5.13 can't host, and VGGT pins numpy<2 against ac-fm's numpy 2.x. Server mode (ADR-server-001) is what lets one hub drive all of them; the env split is ADR-platform-004.

Part 2. Installing the Environments

2.1 Prerequisites & clone

• Linux (tested on Ubuntu 20.04+) with an NVIDIA GPU + recent driver — each env brings its own CUDA runtime, so no system CUDA toolkit is needed.
• Conda or Mamba (Miniforge recommended; the satellite installers use mamba when present).
• Git — a plain clone is enough. The per-env installers clone the simulators / policies into third_party/ at pinned commits for you (there are no git submodules to initialize).

git clone <repo-url> AgentCanvas
cd AgentCanvas

No git submodule step: each install_ac_*.sh clones and pins the third_party/ sources it needs via scripts/install/lib/thirdparty.sh (the single place the upstream commit IDs live). Node.js is not a system prerequisite either — nodejs>=20 ships inside the agentcanvas env.

2.2 The common flow

Every env follows the same shape: one idempotent script per env under scripts/install/ that creates-or-updates a conda env from a project-owned spec. The order is always the same — build the hub first, then add only the satellites you need:

# 1. build the hub first — required by everything else
bash scripts/install/install_core.sh

# 2. add only the satellite env(s) you need (one script per env, idempotent)
bash scripts/install/install_ac_vlnce.sh     # e.g. Habitat
bash scripts/install/install_ac_mp3d.sh      # e.g. MatterSim

Each script: checks prerequisites → creates (or --prune-updates) the env → installs deps, including editable installs from third_party/ → wires up any activation hooks. Re-running a script updates the env in place. The hub script additionally installs the frontend and launches the canvas.

2.3 Per-env specifics

Run the command for each env you need. The Needs column lists what must be present beyond conda + GPU driver; Notes flags anything non-standard.

No env: fetch_ckpt_rt1.sh downloads RT-1-X checkpoints (used from the VLA envs); fetch_dataset_openeqa.sh stages the OpenEQA dataset (reasoner runs in agentcanvas).

2.4 Download data

Simulator evaluations need datasets, scenes, and checkpoints. scripts/data/fetch_data_vlnce.sh fetches them into data/habitat/:

bash scripts/data/fetch_data_vlnce.sh --status              # what you already have
bash scripts/data/fetch_data_vlnce.sh --r2r --mp3d --ddppo  # minimum for VLN-CE (~16GB)
bash scripts/data/fetch_data_vlnce.sh --all                 # everything (ToS prompt for MP3D)

Lands as data/habitat/{datasets, scene_datasets/mp3d, ddppo-models, checkpoints}. Skip entirely for canvas-only / LLM-agent work.

2.5 Run & verify

# 1. launch the canvas (backend :8000 + frontend :5173)
conda activate agentcanvas && bash agentcanvas/launch.sh

# 2. confirm the framework never pulls in a simulator dependency
cd agentcanvas/backend && python -m pytest app/test_import_boundary.py -v

• Backend: http://localhost:8000 (Swagger at /docs) · Frontend: http://localhost:5173
• For auto-reload during development, use bash agentcanvas/run_dev.sh instead of launch.sh.
• LLM keys: copy agentcanvas/.env.example → .env and set AGENTCANVAS_API_KEY / AGENTCANVAS_MODEL / AGENTCANVAS_BASE_URL.