7Batch Evaluation & Job Queue

The capabilities so far cover authoring a graph and watching one run go. This one covers the other half of the research loop: scoring a graph at scale — N episodes from a benchmark split, K worker subprocesses for parallelism, GPU budget gating so concurrent sessions don't OOM each other, and per-episode logs you can ship to a teammate without re-running.

Design docs: Batch Eval System · Server Mode (worker-pool fan-out lives here) · ADRs: eval-001 / eval-002 / eval-003 / eval-004

1. Problem

Single-graph canvas Play answers "does my agent move?". Headline numbers — SR, SPL, nDTW on a 100-/500-episode split — require a different runtime shape:

• Reproducibility: each episode starts from a clean simulator state; failures in episode N don't leak into N+1.
• Parallelism: simulators are slow; K worker subprocesses turn a 6-hour serial run into 1 hour at K=6.
• GPU budget gating: 24 GB of VRAM and three Claude sessions all want to launch eval jobs — somebody has to queue, and the wait can't crash the model in a fourth session.
• Lifetime independent of the backend process: a run that takes 4 hours can't die because the user reloaded the canvas. So each run is its own subprocess.
• Per-episode persistence: when episode 47 reports SR=0 you want to read that episode's log without re-running, and without scanning a 50 MB interleaved JSONL.

2. Architecture — Two Layers

The capability is two cleanly-separated layers stacked together:

The split means the backend stays small and stateless about the run (just a queue entry + a PID + a refcount), while the eval subprocess owns the long-lived state. Note the lifetime coupling is deliberate, and one-directional: a backend restart loses the queue and SIGTERMs running subprocesses (the child arms PR_SET_PDEATHSIG at startup — the no-orphan-VRAM guarantee is bought with restart resilience). What survives is the disk: every episode finished before the kill keeps its ep{NNNN}/ directory, and the next status read of a dead PID without _DONE reports aborted.

3. Run-Dir Layout (ADR-eval-004)

Every run gets a directory at outputs/eval_runs/{run_id}/ where run_id is a second-precision timestamp (e.g. 20260518_143052; collisions get _2, _3, … appended). The layout is shared between the backend (writer of inputs) and the subprocess (writer of outputs):

outputs/eval_runs/{run_id}/
├── spec.json          ← backend writes pre-spawn: full JobSpec + GraphDefinition.to_dict()
├── shared_urls.json   ← backend writes pre-spawn: {nodeset_name: auto_host_url}
├── graph.json         ← BatchEvalRunner writes: run-level graph snapshot
├── summary.json       ← subprocess writes: running snapshot, per-episode rows, terminal
├── _DONE              ← subprocess touches LAST on clean exit (absence ⇒ aborted)
└── episodes/
    └── ep{NNNN}/      ← one self-contained subdir per episode
        ├── log.jsonl     — per-node firing log, never interleaves across workers
        ├── assets/       — image artefacts from this episode
        └── episode.json  — this episode's row of summary.json (self-describing)

Self-containment is the load-bearing property of ADR-eval-004: copy a single ep{NNNN}/ directory to a teammate and they can replay it standalone. The pre-ADR layout interleaved all workers into one root log.jsonl, which made selective replay essentially impossible at worker_count > 1.

4. JobScheduler — Admission & Lifecycle

4.1 Submit

JobScheduler.submit(spec) validates the spec, allocates a fresh run_id, writes spec.json + shared_urls.json into the run dir, and enqueues a _QueuedJob. It does not spawn — admission is the next tick's responsibility, so even a same-second flood of submits is serialised.

spec shape (assembled by api/execution/eval.py):

{
  "eval":     { graph_name, selectors, dataset, split,
                episode_count, step_budget, start_episode_index,
                worker_count, per_step_budget_sec,
                episode_indices, episode_selectors },
  "scheduling": { marginal_vram_mb, exclusive_gpu, priority },
  "graph":    GraphDefinition.to_dict(),
  "_shared_urls": { nodeset_name: auto_host_url },  # also split out to shared_urls.json
  "active_workspace_dir": str | None,                # architect-skill overlay
}

4.2 Tick — admission rule

One tick() per second from the FastAPI lifespan. For each queued job, in priority order:

1. Canvas-lock check: if a canvas Play is running (ExecutionGuard.current()['mode'] == 'canvas'), no new jobs admit. Set via set_canvas_lock_callback. Canvas always wins — preempts even high-priority queued jobs.
2. Budget check: per measurable resource, calibrated estimate ≤ measured_free − Σ reservations (measured admission, 2026-07). marginal_vram_mb is now the declared fallback when calibration is incomplete; already-loaded shared singletons count ~0 in the estimate.
3. Exclusive-GPU jobs: if a queued job has exclusive_gpu=True, it admits only when no other jobs are running. Used by SLAM / VoxPoser-class graphs that don't share GPU memory cleanly.
4. If admitted, spawn the subprocess in its own session (start_new_session=True); the child self-arms PR_SET_PDEATHSIG at startup (eval_subprocess_main.py · _set_pdeathsig) so the kernel reaps it if the backend dies (no orphan VRAM squatting).

Admission has teeth beyond the gate: the scheduler exports the admitted charge as AGENTCANVAS_VRAM_CAP_MB, and the subprocess caps its torch allocator at that figure (eval_subprocess_main.py · _apply_vram_cap) — a run that blows past its estimate OOMs itself instead of its neighbors. Best-effort by design: EGL contexts and child auto_host servers have their own allocators; the measured gate stays the primary defense.

4.3 Cancel + auto-unload

POST /api/eval/v2/runs/{run_id}/cancel from any session terminates the subprocess via SIGTERM → SIGKILL. summary.json stays whatever the subprocess wrote last; absence of _DONE + dead PID is detected by the next status read as aborted.

The scheduler refcounts shared nodesets per consuming job. When a run terminates (clean, aborted, cancelled), each shared nodeset it consumed loses one refcount. Nodesets the scheduler itself loaded for that job (tracked in _shared_loaded_by_jobs) get auto-unloaded at refcount-zero on the next tick. Singletons loaded via canvas Play do not get touched by this path — refcount drops to zero but the entry is not in the auto-unload set.

5. Worker Pool & Batched Inference

These belong topologically to Capability 3 — Isolated Runtime Environments (they're the same subprocess + HTTP plumbing scaled up). The eval-side view is short:

• worker_count = N ⇒ EnvWorkerPool spawns N tagged copies of every env nodeset (parallelism="replicated") and registers RemoteEnvPanelProxy + tagged URL for each. Non-env nodesets stay singleton.
• Each episode leases one WorkerHandle carrying env_panel_overrides + server_url_overrides. The leased LoopRunner consults those before the global registry — so set_episode/play and in-graph tool calls route to that worker's subprocess.
• For pure-functional nodesets (parallelism="shared", e.g. VLM forward) one subprocess hosts a BatchedInferenceServer. K parallel callers' forward() submissions coalesce into one stacked forward pass; K result slices scatter back. Opt-in via batched: ClassVar[bool] = True + batch_dim: ClassVar[str] = "<input_port>" on the node class (ADR-server-003).
• worker_count = 1 is bit-identical to single-worker — empty overrides, no tagged proxies, single shared subprocess for everything.

6. Per-Episode Logs

Each episode runs through a fresh ExecutionLogger that writes to its own episodes/ep{NNNN}/log.jsonl. ExecutionPrinciples.suppress_nav_events = True in batch mode — the WS exec_log event is silenced (it'd be useless: 100 episodes × 14 nodes × 500 steps × 5 sessions would saturate any UI) — but on-disk capture is unchanged.

For the consumer side (replay UI, summary dashboards), summary.json holds one row per episode with {episode_index, episode_id, scene_id, instruction, metrics, step_count, elapsed_sec, status, error, worker_id}. Each episode's episode.json is the same row, written by the subprocess on episode finish so partial summaries are readable even if the run is mid-flight.

7. The Eval Page (GUI)

Everything above also drives from the Evaluate tab. Left column: the submit form — a graph picker whose selection runs POST /introspect to resolve the graph's env nodeset and its selector fields (if that nodeset isn't loaded on this backend, the panel gates on a Load Nodeset button — EnvInfoPanel.tsx), step budget, worker count, Start/Stop; below it, Run History — one row per past run with graph name, status, date, and headline SPL (RunHistory.tsx, re-polled every 3 s while a run is active). Right column: the selected run — status and elapsed time, episode progress bar, per-worker episode chips, aggregate metric cards, a per-episode table with worker attribution, scene, and metrics (EpisodesTable.tsx), and an Export JSON button. Selecting a historical run loads its persisted summary and episode rows via GET /runs/{run_id}; the same run dirs feed the Logs and Replay pages.

8. HTTP API

All routes mounted under /api/eval/v2/ by api/execution/eval.py:

9. The /experiment:run Wrapper

Application code shouldn't talk to the eval API directly when running from a Claude session — go through the /experiment:run <profile> -- <cmd> skill (.claude/experiment/). It:

• Reads marginal_vram_mb / exclusive_gpu from .claude/experiment/profiles.yaml (per-profile, defaults: full GPU + exclusive).
• Allocates a backend port from the 8765–8769 pool, brings up a dedicated auto_host if needed, exports BACKEND_URL=http://127.0.0.1:<port> in the child env.
• The wrapped command reads $BACKEND_URL and submits via the API above. Never hardcode :8000 — that's the user's interactive backend.
• On exit the wrapper releases the admit entry; backend lifetime is owned by /experiment:teardown (no blanket pkill).

See .claude/experiment/README.md for invariants. Bare python tools/eval_*.py / uvicorn / auto_host bypass admission and risk OOM or port collision under concurrent sessions.

10. Key Files

Status