Batch Eval System
Two processes · JobScheduler admission · eval-subprocess cold-start · replicated-spawn vs shared-borrow · per-worker routing
The batch-eval system runs one saved graph across many episodes — optionally fanned out over N parallel workers — and aggregates metrics, persists each run, and streams progress. Today it runs in two processes: a POST /api/eval/v2/start only loads the shared model singletons in the long-lived backend and queues the run; the actual BatchEvalRunner executes in a separate subprocess that JobScheduler spawns once GPU admission clears. Its hard promise is that worker_count is a speed dial, never a result dial — 1 worker and 8 workers score bit-identically. For the rationale see ADR-eval-002 + ADR-server-003; for the surrounding runtime objects see backend-overview + component-registry.
One batch run, two processes. The backend (top) loads the heavy model singletons once and queues the run; JobScheduler spawns an eval subprocess (bottom) that borrows those model servers by URL but spawns and owns its own N tagged env servers. The whole class stack — BatchEvalRunner ⊃ Fleet ⊃ GraphRun ⊃ GraphExecutor — lives inside the subprocess.
1. What the system does
Batch eval answers one question: take a graph the user already drew and ran on the canvas, and run it unattended across many episodes — then report aggregate metrics. The solution has three moves, and the whole page is those three moves traced through code:
- Isolate the run in its own OS process. A crashing episode, a leaked GL context, or a CUDA OOM cannot take the interactive backend down with it, because by default the run executes in a subprocess
JobSchedulerspawned just for it (§ 2–§ 4). - Load each scarce resource once, at the right scope. The heavy model singletons (a VLM, a policy network) stay in the long-lived backend and are borrowed by every run over HTTP; the stateful env simulators are replicated per worker inside the run's own process (§ 4).
- Drive N episodes concurrently. N worker coroutines each pinned to their own env, results merged under a lock (§ 5).
The promise that survives all three moves: worker_count changes how fast the run finishes, never what it scores. Two properties the code is built to keep — both verifiable, both with a cost named in § 5 / § 7:
- Worker count must not change the result. One execution path parameterized by N; at
worker_count=1the pool, the per-worker queues, and the inference coalescer all degenerate to single-tenant rather than taking a separate code path. - Partial results still count. A timed-out or crashed episode still returns whatever metrics + step count were harvested up to the failure (
eval_batch.py · BatchEvalRunner._run_one_episode); failure is just another way an episode finishes.
The smallest run is a single POST — a graph name and an episode count; everything else defaults. On the default path the response is queued, not running: the backend has only enqueued the run, and nothing executes until the scheduler admits it (§ 3).
POST /api/eval/v2/start { "graph_name": "navgpt_mp3d", "episode_count": 10 } # → { "run_id": "20260616_…", "status": "queued", "via_subprocess": true } # default path: the backend only QUEUES the run; nothing executes yet.
2. Two processes, by default
A batch run is split across two OS processes that each load a different half of the world. The request handler start_eval_v2 (api/execution/eval.py · start_eval_v2) branches on one flag:
use_subprocess = req.via_subprocess and os.environ.get("AGENTCANVAS_EVAL_SUBPROCESS", "1") != "0"
| Path | How the backend loads nodesets | Where BatchEvalRunner runs |
Admission gate |
|---|---|---|---|
Subprocess (default, via_subprocess=True) |
ensure_shared_nodesets_for_graph(graph) — loads only shared singletons in the backend; env / replicated nodesets are explicitly skipped (the subprocess owns those). |
In a JobScheduler-spawned subprocess. |
JobScheduler queue + measured resource admission (VRAM + RAM). |
Legacy in-process (via_subprocess=False or AGENTCANVAS_EVAL_SUBPROCESS=0) |
ensure_nodesets_for_graph(graph, worker_count) — loads everything in the backend process. |
In the backend process, as an asyncio task. |
ExecutionGuard single-tenant lock (canvas vs eval). |
The reason for the split is the second move of § 1. A VLM or policy network is many gigabytes and takes minutes to load; running it once per backend and serving every run is the whole point of having a backend. So on the default path the backend only ever loads the shared half: ensure_shared_nodesets_for_graph walks the graph's nodeset prefixes and, for each, continues past anything whose parallelism != "shared" (registry.py · ensure_shared_nodesets_for_graph). The env simulators — stateful, per-call, one-GL-context-per-process — cannot be shared, so they are deliberately not loaded here; the subprocess will spawn its own (§ 4). The backend then computes the URLs of the shared servers it has up and hands them to the subprocess via the spec (§ 3).
Why a second process at all, rather than another asyncio task? Isolation (§ 1, move 1) and admission. The interactive backend serves the canvas; an eval run that segfaults SAPIEN or OOMs the GPU must not take it down, and concurrent runs from sibling sessions must be queued against finite VRAM rather than racing. A separate process gives both for free — the kernel reaps it on crash, and the scheduler meters it before it starts. The legacy in-process path predates the scheduler and survives only for canvas-style debugging, where ExecutionGuard already enforces one-run-at-a-time.
3. Admission & spawn — JobScheduler
submit only enqueues; nothing runs until a later tick admits it. On the subprocess path the handler builds a JobSpec — the EvalConfig, a Scheduling block (marginal_vram_mb, exclusive_gpu, priority), the full graph JSON and an optional active-workspace overlay (services/run_state_io.py) — and calls scheduler.submit(spec, shared_urls=…), the shared-server URL table riding as an argument, not inside the spec (eval.py · start_eval_v2), which returns immediately with {run_id, status: "queued"}.
submit (job_scheduler.py · submit) generates a second-precision run_id (with a collision suffix), creates outputs/eval_runs/{run_id}/, writes spec.json (JobSpec.write) and shared_urls.json, bumps the shared-singleton refcounts, writes an initial summary.json (RunSummary.initial(spec, status="pending")), appends a _QueuedJob, and returns. It does not spawn.
Spawning happens on the scheduler's tick (driven ~1 s by main.py · _scheduler_loop), which runs _reap → _admit → _drain_pending_unloads in that order so VRAM freed by a finished job is available to admission in the same tick. _admit (job_scheduler.py · _admit) walks the queue FIFO and starts the first job that fits:
- Canvas lock. If canvas Play holds
ExecutionGuardin the backend, admission is refused outright (the wired-in_canvas_lock_heldcallback) — the interactive canvas always wins the GPU. - Resource fit. Per measurable resource (VRAM + RAM): calibrated
estimate ≤ measured_free − Σ reservations; a job that doesn't fit is skipped, not blocking. Declaredmarginal_vram_mbis the fallback when calibration is incomplete; the old static ledger survives only for hosts with no sampler. Details: JobScheduler design doc. - Exclusive GPU. An
exclusive_gpujob can't start while anything else runs, and nothing starts while an exclusive job runs. - Priority is stored but ignored. The
priorityfield rides on_QueuedJoband surfaces inlist_active(), but admission is pure FIFO over queue order (the M1 comment in_admit).
When a job fits, _spawn (job_scheduler.py · _spawn) launches the subprocess. The command line is fixed; the spec never travels on argv or stdin (it's read from spec.json), and start_new_session=True puts the child in its own process group so cancel can SIGTERM the whole tree:
subprocess.Popen( [sys.executable, "-m", "app.eval_subprocess_main", "--run-dir", str(run_dir), "--backend-url", self._backend_url], cwd=self._backend_dir, env=env, stdin=subprocess.DEVNULL, start_new_session=True, # setsid → own pgid for clean cancel )
_reap polls each child; on exit it reads the child's final summary.json status, or marks the run aborted if the child died without touching _DONE, then releases the shared-singleton refcounts and drops the job from _running (which is what frees its VRAM reservation). cancel (job_scheduler.py · cancel) removes a still-queued job immediately (status cancelled + _DONE); for a running job it SIGTERMs the process group and lets _reap finalize. JobScheduler itself lives once per process on ProcessServices.job_scheduler (state.py), constructed in the backend's lifespan.
4. The eval subprocess cold-starts its own world
The subprocess is a cold Python process — not a fork — so it re-discovers everything from disk before it can run a single node. eval_subprocess_main.main sets PR_SET_PDEATHSIG (so it and its env children die if the backend dies), parses --run-dir, reads spec.json (JobSpec.read) + shared_urls.json, and builds a fresh WorkspaceComponentRegistry (eval_subprocess_main · _build_local_registry_for_run). Its registry shares nothing with the backend's — get_services() returns a per-process ProcessServices (state.py) — so three steps stand between a cold process and a runnable graph:
registry.scan_all() # cold: re-discover nodesets from disk for name, url in shared_urls.items(): await registry.register_remote_nodeset(name, url) # borrow parent's VLM by URL (no 2nd load) prepared = await prepare_graph(registry, graph, worker_count=worker_count, on_failed="raise", on_unknown="raise")
scan_all()— re-discover from disk. The backend ran this in its lifespan; the subprocess must redo it or the graph's node types resolve to no handler. This is load-bearing: a missing handler is a silent failure, not an error (§ 7). reload-and-code-freshness covers why each subprocess re-scans (and why that's a feature, not a bug — edits are picked up fresh per launch).register_remote_nodeset(name, url)— borrow, don't load. For each shared server the backend has up, the subprocess fetches its manifest over HTTP, generates proxy node classes pointed at the backend's URL, and installs a no-op server shim (registry.py · register_remote_nodeset). The borrowed process is the backend's — the shim'sstop()is intentionally a no-op, so the subprocess exiting never kills the shared VLM. This is how the 14 GB model loads once and serves every run.prepare_graph(registry, graph, worker_count=N, on_failed="raise", on_unknown="raise")— validate, then spawn the env servers it owns.graph_run.py · prepare_graphrunsvalidate_graph_connectivity, callsensure_nodesets_for_graph(graph, worker_count=N)and, with the strict policies the subprocess passes, raisesNodesetLoadErroron any failed or unknown nodeset instead of letting the run drain silently. For each remaining nodeset declaredreplicated, withworker_count > 1, the registry spawns N taggedauto_hostsubprocesses (env_habitat#0 … #N-1) via_load_nodeset_as_server— directpython -m app.server.auto_hostchildren so the one-hopPR_SET_PDEATHSIGreaps them with the run.
The subprocess then builds the EvalRun from spec.eval (the EvalConfig that rode in the JobSpec), hands BatchEvalRunner(run, graph, run_dir=…, on_episode=…) a callback that atomic-rewrites summary.json on every episode boundary (eval_subprocess_main.py · _write_summary → RunSummary.from_run(run).write(run_dir), then get_bus().flush() pushes the episode's events to the parent — this is the live-progress channel for the backend's status poll, since the subprocess has no WebSocket clients, § 6.3), runs BatchEvalRunner.execute(), and in a finally calls shutdown_all() to kill its env servers, save_run, and touch_done (the _DONE marker _reap looks for).
5. Inside one run — the class stack
Everything below this point is identical on both the subprocess and legacy paths — it's the class stack from the diagram, BatchEvalRunner ⊃ Fleet ⊃ GraphRun ⊃ GraphExecutor, plus the shared inference tier the proxy calls reach.
5.1 Fleet — one handle per worker, chunks pinned
The fleet turns the N tagged env servers into N worker handles, and each episode chunk is pinned to one handle. BatchEvalRunner.execute opens a Fleet(graph, worker_count) as an async context manager (components/fleet.py · Fleet). On enter, if worker_count > 1, the fleet builds one WorkerHandle per worker: for every replicated nodeset in the graph it looks up the tagged server URL (registry.get_server_url(name, tag=k)) and, when the nodeset has an env panel, the tagged panel proxy (get_env_panel("env_habitat#k")), and packs them into the handle's server_url_overrides + env_panel_overrides; a tagged copy that is missing is a RuntimeError, not a silent fall-through. At worker_count=1 the overrides are empty and lookups fall through to the global registry — bit-identical to canvas Play. On exit the fleet unloads the tagged copies it routed.
execute then splits the index list into N contiguous chunks (_partition_contiguous, numpy array_split distribution) and gives each worker its own asyncio.Queue; worker k only ever drains queue k and runs every episode on fleet.worker(k) — its own handle, never another worker's. The pinning exists for one reason: a SAPIEN-backed env (SIMPLER) crashes its subprocess after ~15 cross-task set_episode() rebuilds, so each contiguous chunk must stay on one subprocess to bound task switches (eval_batch.py · BatchEvalRunner.execute). N worker coroutines run under one asyncio.gather; they share no step barrier — worker 0 may be on step 8 while worker 1 is on step 2.
5.2 One episode — GraphRun → GraphExecutor
Each episode is a fresh GraphRun — the clean-state guarantee is structural, not a reset path. One iteration of a worker's loop (eval_batch.py · BatchEvalRunner._run_one_episode):
- Set the episode on the worker's tagged env panel: push each cascade field (e.g.
dataset → splitfor Habitat,split → task_idfor SIMPLER) viaon_field_change, pushepisode_index,on_action("play")to commit,on_load()to read back episode id / scene / instruction and the env's per-episodestep_budget(scene-adaptive envs like HM-EQA publish one). - Resolve the step budget: API override → env value → graph value →
DEFAULT_STEP_BUDGET; a resolved budget< 1is refused (it would silently complete a 0-step episode). The wall-clock timeout isresolved_budget × per_step_budget. - Build a fresh
GraphRun(graph, session=RunSession(execution_id, principles=principles), logger=ep_logger, env_panel_overrides=…, server_url_overrides=…)andasyncio.wait_for(graph_run.run(step_budget_override=resolved_budget), timeout). The override carries the resolved budget so parallel workers never mutate the shared graph object. - Harvest metrics from the
RunOutcome(graph_run.py · harvest_metricsreads the executor's graph-scopegraphOutsinks; after a timeout or exceptiongraph_run.outcome()gives the partial one) and build theEpisodeResult— in the success branch and the timeout / exception branches (the partial-results property of § 1). - Evict + aggregate: drop this episode's key from any nodeset-owned keyed container (
_evict_episode_state, worker-safe), then underresults_lockappend the result, recomputeaggregate_metrics+aggregate_by_task, broadcast.
GraphRun builds a GraphExecutor carrying the same overrides, and graph_run.run calls executor.run(...) (graph_run.py · GraphRun.run). The executor resolves its loop bound (step_budget_override → graph.step_budget → DEFAULT) and fires nodes off a ready queue (graph_executor.py · GraphExecutor.run). Two seams matter for eval:
- Per-worker routing. Each in-graph proxy call (e.g.
env_habitat__step_native) asks the executor for a URL override;get_server_url(nodeset)returns this worker's tagged subprocess URL, andserver/proxy.py · _make_forwardpoints itsAutoHostClientthere instead of the URL baked into the proxy class (empty override = baked URL = single-worker / canvas). - Metric harvest. The executor snapshots each
graphOut's pending inputs intonode.state["_last_inputs"]after the terminal iteration (graph_executor.py);graph_run.py · harvest_metricsreads only graph-scopegraphOutsinks (_scope_of(node) == GRAPH_SCOPE_ID), treats aportName == "metrics"value as a dict to flatten, and any other scalar port asmetrics[portName]. Two guards keep a broken run from masquerading as a result: the executor records every{"error": ...}-shaped node result into itsnode_errorsledger and finishes such runs with_status="error", whichRunOutcome.erroredreports —_run_one_episodeconverts that intoepisode.status="error"withoutcome.error_summary()naming the culprit nodes (checked first, since 2026-07-04); and an eval graph that "completed" with no metrics is recorded as an error ("no verdict") so a 0-step drain can't alias tosuccess=0and poison SR.
5.3 Shared inference — one server, one lock
The shared model server takes the K workers' concurrent calls, and a serialized node's one at a time. When K workers hit the same borrowed server, their requests converge server-side; a node that must not run two forwards on its device at once (policy_adapter_vla__predict) sets serialized = True, and the hosting AutoServerApp wraps its handler in the app's single asyncio.Lock (server/auto_server_app.py · _make_handler / _get_inference_lock, created lazily inside the server's running loop). Every other node's calls are concurrent. The contract is pure-functional: no per-caller state, so RNN hidden states must ride explicit hidden_in / hidden_out ports on the wire. (The batched rendezvous of ADR-eval-002 PC — _BatchQueue, flush_timeout_ms, stacked samples — was removed 2026-10-17: its one consumer unrolled every batch sequentially, so the lock was all it provided.)
6. Failure & observability
6.1 Termination paths
Six ways an episode ends, all returning a populated EpisodeResult (eval_batch.py · BatchEvalRunner._run_one_episode):
| Path | status | error | Metrics | step_count |
|---|---|---|---|---|
| Normal completion | completed | None | full | full |
Per-episode timeout (asyncio.wait_for) | error | "timeout after Xs" | partial (last harvest) | up to fail |
| Mid-graph exception | error | str(exc) | partial | up to fail |
| Node error(s) recorded — executor convicts at run end | error | "N node error(s) during run: …" | full (verdict stage ran first) | full |
| Init region failed (e.g. model load, budget<1) | error | str(exc) | empty {} | 0 |
Completed but no graphOut snapshot | error | "no verdict: …" | empty {} | any |
The pool is never blocked by a single stuck episode — each is bounded by asyncio.wait_for at the orchestrator level, and the worker resumes its queue the instant the episode ends. graph_run is pre-bound to None before the try-block so even a failure during GraphRun construction is inspectable.
6.2 Persistence
The run dir is laid out so one episode's trajectory can be reconstructed from its own subdir alone (ADR-eval-004):
outputs/eval_runs/{run_id}/— run_id = timestamp + collision suffix, e.g.20260616_143052spec.json,shared_urls.json— input contract written byJobScheduler.submitbefore spawnsummary.json— run summary; rewritten on every episode boundary in subprocess mode (theon_episodecallback,eval_subprocess_main.py · _write_summary) and once more bysave_runat exitgraph.json— graph frozen atBatchEvalRunnerconstruction (edits to the source graph afterward don't affect the run)stdout.log,stderr.log,_DONE— child output + the terminal marker_reapreadsepisodes/ep{idx:04d}/— one self-contained dir per episodelog.jsonl— this episode's node-firing log onlyassets/— this episode's image artefacts onlyepisode.json— self-describing per-episode record
Active-workspace overlay (ADR-components-009). A run may carry an active_workspace_dir in its spec; JobScheduler._spawn threads it into the subprocess as ACTIVE_WORKSPACE_DIR, so the child's registry overlays workspace/graphs/ + workspace/graph_nodes/ before the frozen base. This is how architect cycles and sibling Claude sessions A/B a graph edit without touching the canonical files. The overlay is resolved at run-start when the graph is frozen; tearing it down afterward doesn't affect the in-flight run.
6.3 Progress: WebSocket vs summary.json
The live-progress channel differs by path, because the WS connection set is per-process. BatchEvalRunner broadcasts eval_progress / eval_episode_done / eval_complete (all tagged source="eval") via state.broadcast, which iterates that process's _ws_connections (state.py · broadcast):
- Legacy in-process path: the runner shares the backend's WS set, so the events reach the UI live.
- Default subprocess path: the subprocess has no WS clients — its
_ws_connectionsis empty — so those broadcasts go nowhere. Live progress instead reaches the UI throughsummary.json, atomic-rewritten on each episode boundary (§ 4) and served by the backend's/api/eval/v2/status+/queueREST endpoints (pull).
Per-node nav_step / exec_log / nav_status events are suppressed during eval either way: BatchEvalRunner.execute sets ExecutionPrinciples(suppress_nav_events=True), and GraphExecutor gates every such broadcast on session.principles.suppress_nav_events (graph_executor.py), so at N workers × base64 images per step the executor never floods the pipe in the first place.
7. Where the code deviates from the mental model
The opening band hands you a clean model: "replicated ⇒ its own subprocess, shared ⇒ borrowed + coalesced, worker_count is free." The shipped code keeps that model only under the conditions below — these are the honest seams.
worker_count=1 does not subprocess a replicated nodeset. ensure_nodesets_for_graph fans out only when worker_count > 1 AND parallelism == "replicated" (registry.py · ensure_nodesets_for_graph). At worker_count=1 even an env nodeset takes the singleton path → load_nodeset(worker_count=1) → an in-process local load unless the class sets server_python. So "replicated ⇒ subprocess" is the model; the code only guarantees it at N>1.
parallelism() for a nodeset that cannot import here. A nodeset whose class could not be imported on the current Python env (its simulator is missing) has no class to read parallelism from, so registry.py · parallelism answers from the declaration the scan read off its source file (declared_nodeset_facts — an AST read of name / parallelism; the framework keeps no table of nodeset names). Only a name the workspace does not declare at all falls back to shared, with a warning — loaded as a singleton K workers share, wrong for an undeclared stateful env.
A missing node handler is a silent no-op, not an error. This is why § 4's scan_all() is load-bearing — if the subprocess skips it, the graph's node types have no handler and every fire returns {}; the episode "completes" at step 0 with empty metrics:
node_cls = CATALOG.get(node.type) if node_cls is None: log.warning("No handler for node type: %s (id=%s)", node.type, node.id) return {} # missing handler → no-op; the episode silently drains at step 0
The "no verdict" guard in § 5.2 / § 6.1 catches the resulting empty-metric completion so it scores as an error rather than success=0. Setting AGENTCANVAS_STRICT_ERRORS additionally re-raises a node's forward() exception out of the loop instead of collapsing it to {"error": ...} (eval/smoke runs use this to fail fast).
Local-mode concurrency at worker_count > 1 is undefined (open gap). A shared / local nodeset that loads weights in-process (no server_python) under N in-process workers has no init lock — init nodes race. The safe path is server mode for any nodeset that loads weights; the in-process race is a known Principle-4 limitation, not a guaranteed behaviour.