AgentCanvas / Pages / Developer Guide / Design Docs / Graph / Batch Eval System
2026-10-17
AgentCanvas / Pages / Developer Guide / Design Docs / Graph / Batch Eval System
2026-07-05

The batch-eval system runs one saved graph across many episodes — optionally fanned out over N parallel workers — and aggregates metrics, persists each run, and streams progress. Today it runs in two processes: a POST /api/eval/v2/start only loads the shared model singletons in the long-lived backend and queues the run; the actual BatchEvalRunner executes in a separate subprocess that JobScheduler spawns once GPU admission clears. Its hard promise is that worker_count is a speed dial, never a result dial — 1 worker and 8 workers score bit-identically. For the rationale see ADR-eval-002 + ADR-server-003; for the surrounding runtime objects see backend-overview + component-registry.

One batch run, two processes. The backend (top) loads the heavy model singletons once and queues the run; JobScheduler spawns an eval subprocess (bottom) that borrows those model servers by URL but spawns and owns its own N tagged env servers. The whole class stack — BatchEvalRunner ⊃ Fleet ⊃ GraphRun ⊃ GraphExecutor — lives inside the subprocess.

Backend process — one, long-lived (the user's :8000) POST /start → ensure_shared_nodesets_for_graph (shared singletons only) → submit(spec) → { queued } JobScheduler queue · FIFO + VRAM admit submit → spec.json tick: reap → admit → spawn WorkspaceComponentRegistry holds shared model singleton handles shared VLM auto_host Prismatic / policy — loads ONCE one inference lock serialized → 1 forward in flight Eval subprocess — one per run (spawned on admission) cold start: scan_all() · register_remote_nodeset(url) · prepare_graph(N) BatchEvalRunner — queue · aggregate · persist Fleet — N WorkerHandles GraphRun — one per episode (fresh) GraphExecutor fires nodes · ready queue graphOut _last_inputs → metrics env auto_host #0 … #N-1 spawned + owned here (PR_SET_PDEATHSIG) replicated nodeset env step/observe → own #k _spawn Popen · start_new_session register_remote_nodeset(url): borrow + route shared calls (coalesced)

1. What the system does

Batch eval answers one question: take a graph the user already drew and ran on the canvas, and run it unattended across many episodes — then report aggregate metrics. The solution has three moves, and the whole page is those three moves traced through code:

  1. Isolate the run in its own OS process. A crashing episode, a leaked GL context, or a CUDA OOM cannot take the interactive backend down with it, because by default the run executes in a subprocess JobScheduler spawned just for it (§ 2–§ 4).
  2. Load each scarce resource once, at the right scope. The heavy model singletons (a VLM, a policy network) stay in the long-lived backend and are borrowed by every run over HTTP; the stateful env simulators are replicated per worker inside the run's own process (§ 4).
  3. Drive N episodes concurrently. N worker coroutines each pinned to their own env, results merged under a lock (§ 5).

The promise that survives all three moves: worker_count changes how fast the run finishes, never what it scores. Two properties the code is built to keep — both verifiable, both with a cost named in § 5 / § 7:

The smallest run is a single POST — a graph name and an episode count; everything else defaults. On the default path the response is queued, not running: the backend has only enqueued the run, and nothing executes until the scheduler admits it (§ 3).

POST /api/eval/v2/start
{ "graph_name": "navgpt_mp3d", "episode_count": 10 }
# → { "run_id": "20260616_…", "status": "queued", "via_subprocess": true }
#   default path: the backend only QUEUES the run; nothing executes yet.

2. Two processes, by default

A batch run is split across two OS processes that each load a different half of the world. The request handler start_eval_v2 (api/execution/eval.py · start_eval_v2) branches on one flag:

use_subprocess = req.via_subprocess and os.environ.get("AGENTCANVAS_EVAL_SUBPROCESS", "1") != "0"
Path How the backend loads nodesets Where BatchEvalRunner runs Admission gate
Subprocess (default, via_subprocess=True) ensure_shared_nodesets_for_graph(graph) — loads only shared singletons in the backend; env / replicated nodesets are explicitly skipped (the subprocess owns those). In a JobScheduler-spawned subprocess. JobScheduler queue + measured resource admission (VRAM + RAM).
Legacy in-process (via_subprocess=False or AGENTCANVAS_EVAL_SUBPROCESS=0) ensure_nodesets_for_graph(graph, worker_count) — loads everything in the backend process. In the backend process, as an asyncio task. ExecutionGuard single-tenant lock (canvas vs eval).

The reason for the split is the second move of § 1. A VLM or policy network is many gigabytes and takes minutes to load; running it once per backend and serving every run is the whole point of having a backend. So on the default path the backend only ever loads the shared half: ensure_shared_nodesets_for_graph walks the graph's nodeset prefixes and, for each, continues past anything whose parallelism != "shared" (registry.py · ensure_shared_nodesets_for_graph). The env simulators — stateful, per-call, one-GL-context-per-process — cannot be shared, so they are deliberately not loaded here; the subprocess will spawn its own (§ 4). The backend then computes the URLs of the shared servers it has up and hands them to the subprocess via the spec (§ 3).

Why a second process at all, rather than another asyncio task? Isolation (§ 1, move 1) and admission. The interactive backend serves the canvas; an eval run that segfaults SAPIEN or OOMs the GPU must not take it down, and concurrent runs from sibling sessions must be queued against finite VRAM rather than racing. A separate process gives both for free — the kernel reaps it on crash, and the scheduler meters it before it starts. The legacy in-process path predates the scheduler and survives only for canvas-style debugging, where ExecutionGuard already enforces one-run-at-a-time.

3. Admission & spawn — JobScheduler

submit only enqueues; nothing runs until a later tick admits it. On the subprocess path the handler builds a JobSpec — the EvalConfig, a Scheduling block (marginal_vram_mb, exclusive_gpu, priority), the full graph JSON and an optional active-workspace overlay (services/run_state_io.py) — and calls scheduler.submit(spec, shared_urls=…), the shared-server URL table riding as an argument, not inside the spec (eval.py · start_eval_v2), which returns immediately with {run_id, status: "queued"}.

submit (job_scheduler.py · submit) generates a second-precision run_id (with a collision suffix), creates outputs/eval_runs/{run_id}/, writes spec.json (JobSpec.write) and shared_urls.json, bumps the shared-singleton refcounts, writes an initial summary.json (RunSummary.initial(spec, status="pending")), appends a _QueuedJob, and returns. It does not spawn.

Spawning happens on the scheduler's tick (driven ~1 s by main.py · _scheduler_loop), which runs _reap → _admit → _drain_pending_unloads in that order so VRAM freed by a finished job is available to admission in the same tick. _admit (job_scheduler.py · _admit) walks the queue FIFO and starts the first job that fits:

When a job fits, _spawn (job_scheduler.py · _spawn) launches the subprocess. The command line is fixed; the spec never travels on argv or stdin (it's read from spec.json), and start_new_session=True puts the child in its own process group so cancel can SIGTERM the whole tree:

subprocess.Popen(
    [sys.executable, "-m", "app.eval_subprocess_main",
     "--run-dir", str(run_dir), "--backend-url", self._backend_url],
    cwd=self._backend_dir, env=env, stdin=subprocess.DEVNULL,
    start_new_session=True,  # setsid → own pgid for clean cancel
)

_reap polls each child; on exit it reads the child's final summary.json status, or marks the run aborted if the child died without touching _DONE, then releases the shared-singleton refcounts and drops the job from _running (which is what frees its VRAM reservation). cancel (job_scheduler.py · cancel) removes a still-queued job immediately (status cancelled + _DONE); for a running job it SIGTERMs the process group and lets _reap finalize. JobScheduler itself lives once per process on ProcessServices.job_scheduler (state.py), constructed in the backend's lifespan.

4. The eval subprocess cold-starts its own world

The subprocess is a cold Python process — not a fork — so it re-discovers everything from disk before it can run a single node. eval_subprocess_main.main sets PR_SET_PDEATHSIG (so it and its env children die if the backend dies), parses --run-dir, reads spec.json (JobSpec.read) + shared_urls.json, and builds a fresh WorkspaceComponentRegistry (eval_subprocess_main · _build_local_registry_for_run). Its registry shares nothing with the backend's — get_services() returns a per-process ProcessServices (state.py) — so three steps stand between a cold process and a runnable graph:

registry.scan_all()                                    # cold: re-discover nodesets from disk
for name, url in shared_urls.items():
    await registry.register_remote_nodeset(name, url)  # borrow parent's VLM by URL (no 2nd load)
prepared = await prepare_graph(registry, graph, worker_count=worker_count, on_failed="raise", on_unknown="raise")
  1. scan_all() — re-discover from disk. The backend ran this in its lifespan; the subprocess must redo it or the graph's node types resolve to no handler. This is load-bearing: a missing handler is a silent failure, not an error (§ 7). reload-and-code-freshness covers why each subprocess re-scans (and why that's a feature, not a bug — edits are picked up fresh per launch).
  2. register_remote_nodeset(name, url) — borrow, don't load. For each shared server the backend has up, the subprocess fetches its manifest over HTTP, generates proxy node classes pointed at the backend's URL, and installs a no-op server shim (registry.py · register_remote_nodeset). The borrowed process is the backend's — the shim's stop() is intentionally a no-op, so the subprocess exiting never kills the shared VLM. This is how the 14 GB model loads once and serves every run.
  3. prepare_graph(registry, graph, worker_count=N, on_failed="raise", on_unknown="raise") — validate, then spawn the env servers it owns. graph_run.py · prepare_graph runs validate_graph_connectivity, calls ensure_nodesets_for_graph(graph, worker_count=N) and, with the strict policies the subprocess passes, raises NodesetLoadError on any failed or unknown nodeset instead of letting the run drain silently. For each remaining nodeset declared replicated, with worker_count > 1, the registry spawns N tagged auto_host subprocesses (env_habitat#0 … #N-1) via _load_nodeset_as_server — direct python -m app.server.auto_host children so the one-hop PR_SET_PDEATHSIG reaps them with the run.

The subprocess then builds the EvalRun from spec.eval (the EvalConfig that rode in the JobSpec), hands BatchEvalRunner(run, graph, run_dir=…, on_episode=…) a callback that atomic-rewrites summary.json on every episode boundary (eval_subprocess_main.py · _write_summary → RunSummary.from_run(run).write(run_dir), then get_bus().flush() pushes the episode's events to the parent — this is the live-progress channel for the backend's status poll, since the subprocess has no WebSocket clients, § 6.3), runs BatchEvalRunner.execute(), and in a finally calls shutdown_all() to kill its env servers, save_run, and touch_done (the _DONE marker _reap looks for).

5. Inside one run — the class stack

Everything below this point is identical on both the subprocess and legacy paths — it's the class stack from the diagram, BatchEvalRunner ⊃ Fleet ⊃ GraphRun ⊃ GraphExecutor, plus the shared inference tier the proxy calls reach.

5.1 Fleet — one handle per worker, chunks pinned

The fleet turns the N tagged env servers into N worker handles, and each episode chunk is pinned to one handle. BatchEvalRunner.execute opens a Fleet(graph, worker_count) as an async context manager (components/fleet.py · Fleet). On enter, if worker_count > 1, the fleet builds one WorkerHandle per worker: for every replicated nodeset in the graph it looks up the tagged server URL (registry.get_server_url(name, tag=k)) and, when the nodeset has an env panel, the tagged panel proxy (get_env_panel("env_habitat#k")), and packs them into the handle's server_url_overrides + env_panel_overrides; a tagged copy that is missing is a RuntimeError, not a silent fall-through. At worker_count=1 the overrides are empty and lookups fall through to the global registry — bit-identical to canvas Play. On exit the fleet unloads the tagged copies it routed.

execute then splits the index list into N contiguous chunks (_partition_contiguous, numpy array_split distribution) and gives each worker its own asyncio.Queue; worker k only ever drains queue k and runs every episode on fleet.worker(k) — its own handle, never another worker's. The pinning exists for one reason: a SAPIEN-backed env (SIMPLER) crashes its subprocess after ~15 cross-task set_episode() rebuilds, so each contiguous chunk must stay on one subprocess to bound task switches (eval_batch.py · BatchEvalRunner.execute). N worker coroutines run under one asyncio.gather; they share no step barrier — worker 0 may be on step 8 while worker 1 is on step 2.

5.2 One episode — GraphRun → GraphExecutor

Each episode is a fresh GraphRun — the clean-state guarantee is structural, not a reset path. One iteration of a worker's loop (eval_batch.py · BatchEvalRunner._run_one_episode):

  1. Set the episode on the worker's tagged env panel: push each cascade field (e.g. dataset → split for Habitat, split → task_id for SIMPLER) via on_field_change, push episode_index, on_action("play") to commit, on_load() to read back episode id / scene / instruction and the env's per-episode step_budget (scene-adaptive envs like HM-EQA publish one).
  2. Resolve the step budget: API override → env value → graph value → DEFAULT_STEP_BUDGET; a resolved budget < 1 is refused (it would silently complete a 0-step episode). The wall-clock timeout is resolved_budget × per_step_budget.
  3. Build a fresh GraphRun(graph, session=RunSession(execution_id, principles=principles), logger=ep_logger, env_panel_overrides=…, server_url_overrides=…) and asyncio.wait_for(graph_run.run(step_budget_override=resolved_budget), timeout). The override carries the resolved budget so parallel workers never mutate the shared graph object.
  4. Harvest metrics from the RunOutcome (graph_run.py · harvest_metrics reads the executor's graph-scope graphOut sinks; after a timeout or exception graph_run.outcome() gives the partial one) and build the EpisodeResult — in the success branch and the timeout / exception branches (the partial-results property of § 1).
  5. Evict + aggregate: drop this episode's key from any nodeset-owned keyed container (_evict_episode_state, worker-safe), then under results_lock append the result, recompute aggregate_metrics + aggregate_by_task, broadcast.

GraphRun builds a GraphExecutor carrying the same overrides, and graph_run.run calls executor.run(...) (graph_run.py · GraphRun.run). The executor resolves its loop bound (step_budget_override → graph.step_budget → DEFAULT) and fires nodes off a ready queue (graph_executor.py · GraphExecutor.run). Two seams matter for eval:

5.3 Shared inference — one server, one lock

The shared model server takes the K workers' concurrent calls, and a serialized node's one at a time. When K workers hit the same borrowed server, their requests converge server-side; a node that must not run two forwards on its device at once (policy_adapter_vla__predict) sets serialized = True, and the hosting AutoServerApp wraps its handler in the app's single asyncio.Lock (server/auto_server_app.py · _make_handler / _get_inference_lock, created lazily inside the server's running loop). Every other node's calls are concurrent. The contract is pure-functional: no per-caller state, so RNN hidden states must ride explicit hidden_in / hidden_out ports on the wire. (The batched rendezvous of ADR-eval-002 PC — _BatchQueue, flush_timeout_ms, stacked samples — was removed 2026-10-17: its one consumer unrolled every batch sequentially, so the lock was all it provided.)

6. Failure & observability

6.1 Termination paths

Six ways an episode ends, all returning a populated EpisodeResult (eval_batch.py · BatchEvalRunner._run_one_episode):

PathstatuserrorMetricsstep_count
Normal completioncompletedNonefullfull
Per-episode timeout (asyncio.wait_for)error"timeout after Xs"partial (last harvest)up to fail
Mid-graph exceptionerrorstr(exc)partialup to fail
Node error(s) recorded — executor convicts at run enderror"N node error(s) during run: …"full (verdict stage ran first)full
Init region failed (e.g. model load, budget<1)errorstr(exc)empty {}0
Completed but no graphOut snapshoterror"no verdict: …"empty {}any

The pool is never blocked by a single stuck episode — each is bounded by asyncio.wait_for at the orchestrator level, and the worker resumes its queue the instant the episode ends. graph_run is pre-bound to None before the try-block so even a failure during GraphRun construction is inspectable.

6.2 Persistence

The run dir is laid out so one episode's trajectory can be reconstructed from its own subdir alone (ADR-eval-004):

Active-workspace overlay (ADR-components-009). A run may carry an active_workspace_dir in its spec; JobScheduler._spawn threads it into the subprocess as ACTIVE_WORKSPACE_DIR, so the child's registry overlays workspace/graphs/ + workspace/graph_nodes/ before the frozen base. This is how architect cycles and sibling Claude sessions A/B a graph edit without touching the canonical files. The overlay is resolved at run-start when the graph is frozen; tearing it down afterward doesn't affect the in-flight run.

6.3 Progress: WebSocket vs summary.json

The live-progress channel differs by path, because the WS connection set is per-process. BatchEvalRunner broadcasts eval_progress / eval_episode_done / eval_complete (all tagged source="eval") via state.broadcast, which iterates that process's _ws_connections (state.py · broadcast):

Per-node nav_step / exec_log / nav_status events are suppressed during eval either way: BatchEvalRunner.execute sets ExecutionPrinciples(suppress_nav_events=True), and GraphExecutor gates every such broadcast on session.principles.suppress_nav_events (graph_executor.py), so at N workers × base64 images per step the executor never floods the pipe in the first place.

7. Where the code deviates from the mental model

The opening band hands you a clean model: "replicated ⇒ its own subprocess, shared ⇒ borrowed + coalesced, worker_count is free." The shipped code keeps that model only under the conditions below — these are the honest seams.

worker_count=1 does not subprocess a replicated nodeset. ensure_nodesets_for_graph fans out only when worker_count > 1 AND parallelism == "replicated" (registry.py · ensure_nodesets_for_graph). At worker_count=1 even an env nodeset takes the singleton path → load_nodeset(worker_count=1) → an in-process local load unless the class sets server_python. So "replicated ⇒ subprocess" is the model; the code only guarantees it at N>1.

parallelism() for a nodeset that cannot import here. A nodeset whose class could not be imported on the current Python env (its simulator is missing) has no class to read parallelism from, so registry.py · parallelism answers from the declaration the scan read off its source file (declared_nodeset_facts — an AST read of name / parallelism; the framework keeps no table of nodeset names). Only a name the workspace does not declare at all falls back to shared, with a warning — loaded as a singleton K workers share, wrong for an undeclared stateful env.

A missing node handler is a silent no-op, not an error. This is why § 4's scan_all() is load-bearing — if the subprocess skips it, the graph's node types have no handler and every fire returns {}; the episode "completes" at step 0 with empty metrics:

node_cls = CATALOG.get(node.type)
if node_cls is None:
    log.warning("No handler for node type: %s (id=%s)", node.type, node.id)
    return {}   # missing handler → no-op; the episode silently drains at step 0

The "no verdict" guard in § 5.2 / § 6.1 catches the resulting empty-metric completion so it scores as an error rather than success=0. Setting AGENTCANVAS_STRICT_ERRORS additionally re-raises a node's forward() exception out of the loop instead of collapsing it to {"error": ...} (eval/smoke runs use this to fail fast).

Local-mode concurrency at worker_count > 1 is undefined (open gap). A shared / local nodeset that loads weights in-process (no server_python) under N in-process workers has no init lock — init nodes race. The safe path is server mode for any nodeset that loads weights; the in-process race is a known Principle-4 limitation, not a guaranteed behaviour.

AgentCanvas docs