AgentCanvas / Pages / Developer Guide / Core / Decisions / Eval / ADR-eval-003
2026-05-07 22:55

ADR-eval-003

Subprocess-per-run job scheduler with cross-session admission

Date
2026-05-07 22:55
Status
accepted
Field
eval

Context

Eval Phase 2 (ADR-eval-002) shipped a working worker pool inside one in-process BatchEvalRunner (agent_loop/eval_batch.py:222), driven from a single FastAPI handler that takes the singleton ExecutionGuard.eval mutex (api/execution/eval.py:176) and dispatches as asyncio.create_task(_run_and_persist()) (api/execution/eval.py:228). The mutex is per-backend-process β€” at most one batch eval runs at a time per backend. To work around it, every Claude session has been starting its own uvicorn app.main:app on a port from 8765–8769 via the /experiment:run skill harness (.claude/commands/experiment/run.md pre-2026-05-07), with a parallel admission-control system in .claude/experiment/bin/admit.py reading and writing JSON files under .claude/experiment/runtime/ to coordinate between sessions. Two failure modes have surfaced repeatedly: (i) GPU OOM because each backend loads its own copy of Prismatic VLM (~14 GB), so two concurrent sessions blow a 24 GB card even when neither's eval is heavy β€” the JSON-based admission ledger has no way to amortize a shared singleton across sessions because every session owns a different WorkspaceComponentRegistry; (ii) cross-session backend kills, where one session's PGID-scoped cleanup trap fires at the wrong moment and another session's running eval dies with it. The skill's "no blanket pkill" memory (feedback_architect_cleanup) was added after one such incident on 2026-05-07. Both failure modes share one root cause: backends are per-session ephemeral, but the scarce resources they manage (VRAM, env subprocesses, model checkpoints) are per-host. The architecture has the wrong unit of scope.

Decision

One long-lived agentcanvas backend per host. Every eval submitted via HTTP runs as a separate Python subprocess owned by an admission-controlled scheduler inside the backend. Six rules.

(1) JobScheduler service inside the backend (agentcanvas/backend/app/services/job_scheduler.py). Singleton attached to ProcessServices, created in FastAPI lifespan. Owns: a FIFO queue of _QueuedJob, a dict of _RunningJob, a VRAM ledger (usable_vram_mb βˆ’ Ξ£ active.marginal_vram_mb), and a per-second tick() that admits queued jobs whose declared marginal fits and reaps finished Popen children. Cross-session admission is automatic because the backend is shared: every Claude session that POSTs /api/eval/v2/start to the same backend hits the same scheduler, the same queue, the same ledger. GET /api/eval/v2/queue is the cross-session view.

(2) Run subprocess as the unit of execution (agentcanvas/backend/app/eval_subprocess_main.py). Admit β†’ Popen([python, -m, app.eval_subprocess_main, --run-dir, /abs/path], setsid+PR_SET_PDEATHSIG). The subprocess reads spec.json + shared_urls.json from the run dir, registers the parent's shared singletons via WorkspaceComponentRegistry.register_remote_nodeset(name, url) (no second 14 GB load), spawns its own tagged env-worker subprocesses via the existing ensure_nodesets_for_graph(worker_count=N) path, calls BatchEvalRunner.execute() unmodified, writes summary.json + _DONE, exits. Fault isolation falls out of the OS process boundary: a run subprocess crash (segfault, CUDA OOM raised as Python, SIGKILL from kernel OOM-killer) cannot bring down the parent backend or sibling runs. Code-reload-friendly dev iteration falls out for free: each run is a fresh Python interpreter, picks up backend code edits without backend restart.

(3) Disk is the IPC channel; no schedulerβ†’subprocess RPC. Backend writes spec.json + shared_urls.json pre-spawn; subprocess writes summary.json (atomic via tempfile + os.replace with pid+tid suffix to avoid concurrent-writer races) on every episode boundary and on terminal exit; subprocess touches _DONE last. Backend status queries (GET /api/eval/v2/runs/{id}) are file reads, not RPC. _DONE presence ⇔ status was finalized; absence + dead PID = "aborted" (Q1 fault model). Backend startup walks outputs/eval_runs/, runs reconcile_aborted_runs to flip stale running β†’ aborted for crashed runs from previous backend lifetimes. This is deliberately not a JSON-RPC channel because (a) the live state already needs to land on disk for restart resilience anyway, (b) RPC adds a failure mode (subprocess deadlocks on a backend that's not reading) that file-based comms doesn't have.

(4) Q1 fault model β€” in-flight loss accepted. When the backend dies (user restart, OOM, segfault, kill -9), every running run subprocess receives SIGTERM via PR_SET_PDEATHSIG and dies with it. No checkpointing of in-flight runs, no resume-on-restart, no per-run independent supervisor process. Persisted records carry _DONE (presence β‡’ clean terminal); absence after the parent goes away = the user manually decides whether to re-submit or sweep. The decision is calibrated to the typical restart cause (developer changed code and wants the new code to take effect) β€” at that moment the running episodes are using soon-to-be-stale code anyway, so losing them is honest. The cost of the simpler model (no run-supervisor process, no checkpoint protocol, no resume-after-restart code path) outweighs the cost of one episode's lost work.

(5) Q2 resource accounting β€” singleton baseline implicit, marginal declared. Shared singletons (Prismatic VLM, BLIP-2 caption, etc.) are loaded once by the backend at first-use and stay live; their VRAM footprint is implicit in the difference between nvidia-smi total and AGENTCANVAS_USABLE_VRAM_MB (default total - 16000, configurable per host). Each job spec declares scheduling.marginal_vram_mb for what its env-side workers + per-job adapters add on top. Admission rule: Ξ£ active.marginal ≀ usable. No live nvidia-smi polling, no double-counting; if the operator declares wrong, the run OOMs and gets handled by Q1. This is a deliberate trust-the-author choice over reserve-then-measure, mirroring how SLURM partition memory works: declarations are policy, not predictions.

(6) /api/eval/v2/start gains via_subprocess (default True), marginal_vram_mb, exclusive_gpu, priority. Default-on flips the backend's eval semantics from "in-process asyncio task gated by ExecutionGuard.eval" to "subprocess gated by JobScheduler.admit". via_subprocess=false (or env AGENTCANVAS_EVAL_SUBPROCESS=0) keeps the legacy path alive as an emergency-rollback escape hatch. ExecutionGuard.canvas mode is unchanged β€” canvas Play vs eval mutex still exists, scheduler's _canvas_lock_held callback respects it. ExecutionGuard.eval mode survives only as the legacy path's lock; once the legacy path is removed (future ADR), eval mutex retires entirely and N concurrent runs become the only mode.

Out of scope (deferred): (i) Multi-host federation β€” scheduler is per-host today; cross-host job submission would need a registration/heartbeat protocol that's a separate ADR. (ii) Frontend Eval-page status compatibility β€” /api/eval/v2/status and /episodes still read the legacy _current_run, returning "none" for subprocess runs; UI fix lands in a follow-up. (iii) WebSocket event forwarding from subprocess to backend so the canvas can stream live progress β€” M2 work, currently subprocess writes only to disk and frontend can poll /runs/{id} for state. (iv) Priority lanes / preemption β€” M3 admission today is FIFO. (v) Job dependencies (depends_on) β€” out of M1; trivial extension to admission rule when needed.

Alternatives

(a) Keep one backend, allow N concurrent in-process eval tasks via per-eval ExecutionGuard.eval#K β€” rejected: the executor isn't fully reentrant against shared in-process state (one buggy run's uncaught exception in graph_executor can bring down the asyncio loop and every sibling run with it), memory leaks accumulate across runs in one Python heap, and dev iteration needs a backend restart to pick up code edits which then loses every running run anyway β€” same blast radius as subprocess approach without any of its benefits. (b) Cgroups + process-namespacing per run, sharing one Python interpreter via os.fork β€” rejected: post-fork CUDA state corrupts on most drivers, requires a "no GPU init before fork" invariant that breaks the moment any imported module touches torch.cuda, and the lifecycle complexity (per-run cgroup teardown, namespace cleanup) is bigger than just calling subprocess.Popen. (c) Per-run independent supervisor process that survives backend restart (resume in-flight on next backend boot via reading checkpoints) β€” rejected: episode-level checkpointing of EvalRun state requires a serialization protocol the framework doesn't have, the supervisor adds a third process tier (backend ↔ supervisor ↔ run), and the realistic cause of restart is "developer changed code" β€” at which point the in-flight run was using stale code and recovering it has negative value. (d) JSON-RPC channel from subprocess to backend for live state instead of file-based summary.json β€” rejected: (i) live state must land on disk anyway for restart-resume of queued jobs and for cross-session GET /runs/{id} after the originating run subprocess exits, so RPC duplicates the protocol; (ii) RPC introduces a deadlock vector (subprocess blocks on a backend that's not reading); (iii) atomic file rename is cheap (sub-ms at episode cadence). (e) Separate tools/exp/ daemon outside the backend managing admission via SQLite + spawning per-job uvicorn subprocesses (the design that was prototyped and deleted on 2026-05-07) β€” rejected: the agentcanvas backend already has every primitive the daemon would re-implement (FastAPI server, WorkspaceComponentRegistry, eval pipeline, run state store), and per-job uvicorn replicates the original "every session loads its own Prismatic" anti-pattern.

Rationale

The trampling problem this ADR solves is one of resource scope mismatch β€” backends were ephemeral but the resources they managed (VRAM, model checkpoints, env subprocesses) are per-host. Moving the backend to host-scope and the eval to per-run-subprocess-scope realigns the layers: each tier owns the lifetime of resources at its scale. The JobScheduler inside the backend is structurally simpler than the deleted external tools/exp/ because it reuses what the backend already has β€” WorkspaceComponentRegistry for nodeset URLs, outputs/eval_runs/ for run state, FastAPI for the API surface, lifespan for the tick loop. The disk-as-IPC choice (rule 3) is load-bearing for restart resilience: every other state mechanism (JSON-RPC, in-memory dicts, SQLite) needs special-case recovery code; atomic file rename is already restart-safe by construction. The Q1 fault model (rule 4) accepts a precisely-scoped loss class (in-flight runs at backend death) in exchange for not building a checkpoint/resume/supervisor stack β€” a 10Γ— simplicity win for a real but bounded cost. The Q2 model (rule 5) trusts operators to declare correctly because it has to: live nvidia-smi polling double-counts (a job's declared marginal would be subtracted from free, but then the actual model load would also reduce free, blocking sibling admissions), and the OOM blast radius is contained by Q1. The default-on flip in rule 6 forces the new path through the live frontend without a feature-flag dance β€” frontend's /start POST without explicit scheduling block defaults to marginal_vram_mb=0, priority=normal, treated as a CPU-only task and admitted immediately, which is bit-identical to the legacy path's "always admit" behaviour for interactive runs. The ExecutionGuard.eval legacy lock survives intentionally β€” it lets the rollback escape hatch (via_subprocess=false) keep working without a bigger refactor, and pruning it can be a one-line deletion in a follow-up ADR once subprocess path is the sole entrypoint. Together: M1 ships a strictly better default behaviour without breaking any callers, and the things it doesn't ship (multi-host, WS streaming, priority) are clean extension points rather than baked-in compromises.

Affected docs

Follow-up: A future ADR retires ExecutionGuard.eval mode entirely once via_subprocess=false legacy path is deleted, and extends JobScheduler to multi-host (worker registration + heartbeat). WS event forwarding from subprocess to backend is M2 work, no ADR needed.