Installation

AgentCanvas is not one environment — it is one orchestrator hub (the agentcanvas conda env) surrounded by a fleet of satellite envs, one per simulator or model. Part 1 is the mental model: what envs exist and how they relate. Part 2 is the mechanics: how to install each one and the flow they share.

Part 1. Mental Model

Everything runs through the agentcanvas env: it hosts the backend, the frontend, the graph executor, every method / reasoning nodeset (NavGPT, MapGPT, AO-Planner, …), and all LLM calls. It deliberately never imports a simulator. Each heavy dependency that can't coexist with that stack — Habitat-Sim 0.1.7 (pinned to Python 3.8), MatterSim (its own C++ build), the VLA policy stacks (their own torch/JAX), DetAny3D (CUDA 11.8) — lives in its own satellite env and is reached over server mode (the backend auto-hosts it as a subprocess and talks to it over HTTP). So the picture is hub-and-spoke:

The same picture as a list. One hub + eleven satellite envs. Two installers create no env at all: fetch_ckpt_rt1.sh just downloads RT-1-X checkpoints (used from the VLA envs) and fetch_dataset_openeqa.sh just stages the OpenEQA dataset (the EM-EQA reasoner runs inside agentcanvas with gpt-4o). agentcanvas-test is a throwaway clone of the hub for CI.

Why so many. Two forces: (1) the framework import boundary — the backend in agentcanvas/backend/app/ may never import habitat, habitat_sim, vlnce_baselines, or habitat_baselines; and (2) mutually incompatible pins — Habitat-Sim 0.1.7 needs Python 3.8, DetAny3D needs CUDA 11.8, the VLA stacks need their own torch/JAX, and the frozen FM research stacks (Prismatic, RAM, SpatialBot) pin old transformers that the shared ac-fm env's 5.13 can't host, and VGGT pins numpy<2 against ac-fm's numpy 2.x. Server mode (ADR-server-001) is what lets one hub drive all of them; the env split is ADR-platform-004.

Part 2. Installing the Environments

2.1 Prerequisites & clone

• Linux (tested on Ubuntu 20.04+) with an NVIDIA GPU + recent driver — each env brings its own CUDA runtime, so no system CUDA toolkit is needed.
• Conda or Mamba (Miniforge recommended; the satellite installers use mamba when present).
• Git — a plain clone is enough. The per-env installers clone the simulators / policies into third_party/ at pinned commits for you (there are no git submodules to initialize).
• Rootless Docker — only if you will run a Container-Launch nodeset (model_orbslam3, model_pyslam). Not needed for anything else; setup in §2.6.

git clone <repo-url> AgentCanvas
cd AgentCanvas

No git submodule step: each install_ac_*.sh clones and pins the third_party/ sources it needs via scripts/install/lib/thirdparty.sh (the single place the upstream commit IDs live). Node.js is not a system prerequisite either — nodejs>=20 ships inside the agentcanvas env.

2.2 The common flow

Every env follows the same shape: one idempotent script per env under scripts/install/ that creates-or-updates a conda env from a project-owned spec. The order is always the same — build the hub first, then add only the satellites you need:

# 1. build the hub first — required by everything else
bash scripts/install/install_core.sh

# 2. add only the satellite env(s) you need (one script per env, idempotent)
bash scripts/install/install_ac_vlnce.sh     # e.g. Habitat
bash scripts/install/install_ac_mp3d.sh      # e.g. MatterSim

Each script: checks prerequisites → creates (or --prune-updates) the env → installs deps, including editable installs from third_party/ → wires up any activation hooks. Re-running a script updates the env in place. The hub script additionally installs the frontend and launches the canvas.

2.3 Per-env specifics

Run the command for each env you need. The Needs column lists what must be present beyond conda + GPU driver; Notes flags anything non-standard.

No env: fetch_ckpt_rt1.sh downloads RT-1-X checkpoints (used from the VLA envs); fetch_dataset_openeqa.sh stages the OpenEQA dataset (reasoner runs in agentcanvas).

2.4 Download data

Simulator evaluations need datasets, scenes, and checkpoints. scripts/data/fetch_data_vlnce.sh fetches them into data/habitat/:

bash scripts/data/fetch_data_vlnce.sh --status              # what you already have
bash scripts/data/fetch_data_vlnce.sh --r2r --mp3d --ddppo  # minimum for VLN-CE (~16GB)
bash scripts/data/fetch_data_vlnce.sh --all                 # everything (ToS prompt for MP3D)

Lands as data/habitat/{datasets, scene_datasets/mp3d, ddppo-models, checkpoints}. Skip entirely for canvas-only / LLM-agent work.

2.5 Run & verify

# 1. launch the canvas (backend :8000 + frontend :5173)
conda activate agentcanvas && bash agentcanvas/launch.sh

# 2. confirm the framework never pulls in a simulator dependency
cd agentcanvas/backend && python -m pytest app/test_import_boundary.py -v

• Backend: http://localhost:8000 (Swagger at /docs) · Frontend: http://localhost:5173
• For auto-reload during development, use bash agentcanvas/run_dev.sh instead of launch.sh.
• LLM keys: copy agentcanvas/.env.example → .env and set AGENTCANVAS_API_KEY / AGENTCANVAS_MODEL / AGENTCANVAS_BASE_URL.

2.6 Rootless Docker (Container-Launch nodesets only)

Some nodesets skip conda entirely and run inside a Docker container — Container Launch (design doc, ADR-server-005). Today that is model_orbslam3 and model_pyslam; if you don't use them, skip this section. The framework talks only to a rootless daemon on your user socket (unix:///run/user/<uid>/docker.sock — hardcoded in ContainerServer, the system daemon at /var/run/docker.sock is never used), so the daemon must be set up per-user once. Root privileges are needed only for the two apt installs below; everything the daemon does afterwards runs as you.

1. Engine + user daemon. Install the engine packages, then create the per-user daemon (upstream rootless guide):

# one-time, needs sudo — engine + rootless extras (docs.docker.com/engine/install/ubuntu)
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-ce-rootless-extras uidmap

# subordinate-id ranges must exist for your user (installers usually add them)
grep "$USER" /etc/subuid /etc/subgid        # expect <user>:100000:65536

# per-user from here on, no sudo — create and start the user daemon
dockerd-rootless-setuptool.sh install
systemctl --user enable --now docker
docker context use rootless

2. Daemon config — big-disk data-root + CDI. Server images are large (the pySLAM image is ~55 GB); point data-root at a disk with room before pulling anything, and enable CDI while you're in the file (it is how GPUs reach rootless containers, step 3). Adjust the path to your machine:

mkdir -p ~/.config/docker
cat > ~/.config/docker/daemon.json <<'EOF'
{
  "data-root": "/data/docker-rootless",
  "features": { "cdi": true }
}
EOF
systemctl --user restart docker             # safe while no containers run

3. GPU via CDI. The legacy --gpus all flag fails under rootless Docker on cgroup-v1 hosts (it cannot write device rules); the working path is CDI, which the framework already uses — a nodeset with server_image_gpu = True is started with --device nvidia.com/gpu=all and degrades to CPU once if the GPU start fails.

# one-time, needs sudo — the toolkit that writes CDI specs
sudo apt-get install nvidia-container-toolkit

# user-level — declare the GPUs to the daemon (rerun after a driver upgrade)
mkdir -p ~/.config/cdi
nvidia-ctk cdi generate --output=$HOME/.config/cdi/nvidia.yaml

4. Verify.

docker run --rm hello-world                                   # daemon works
docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi # GPU visible via CDI
sudo loginctl enable-linger "$USER"   # optional: keep the daemon alive after logout

• Without enable-linger, the daemon stops when you fully log out; after a reboot, systemctl --user start docker brings it back.
• The backend forces DOCKER_HOST to the user socket itself, so canvas runs need no environment variable. For manual docker build/run from a non-login shell, export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock first.
• Server images are built locally, never auto-pulled: each Container-Launch nodeset ships its recipe, e.g. bash workspace/nodesets/model/model_orbslam3/docker/build.sh. Loading a nodeset whose image is missing fails with an explicit "build it" error.