JobScheduler

The JobScheduler answers one question: is it safe to start this eval run right now? When you submit a run, nothing executes yet — the job waits in a queue. Once a second, the scheduler looks at how much GPU and system memory is actually free, estimates how much this job will need (learned from watching its previous runs), and starts it only if it fits. Everything it used to make that call is visible in one API response (GET /api/eval/v2/queue), so when a job doesn't start, you can always see why. For what happens inside a run, see batch-eval; for how model server processes work, see server-mode.

The whole system is one loop. A submitted job waits in the queue until the gate says it fits. While it runs, its memory is measured; when it ends, the measurements are saved (sediment). The saved measurements are where the gate's estimate comes from — and the live measurements are where free comes from.

1. What it does

One machine runs many eval jobs at once, and none of them know about each other. If two big jobs start together, the GPU runs out of memory and both crash. The scheduler is the single place where every job must ask permission first.

An earlier version asked each job to declare how much GPU memory it would use — and trusted the number. Declarations were routinely wrong (the default was 0), so the gate protected nothing. The current version measures instead: every second, each process's GPU memory and RAM is read and attributed to an owner (§4.1), and admission checks per resource: does this job's estimated need fit into what is actually free? A machine without a GPU has no VRAM dimension to check — it gates on RAM alone.

The estimates are learned, not configured (resource_stats.py · CalibrationStore): the first time a graph runs, the scheduler just watches it, and writes what it saw into a calibration file when the job finishes — we call this sedimenting. From the second run on, admission uses the measured numbers, keyed to a hash of the nodeset's source code so edits retire stale statistics.

An admitted run executes in its own OS process (app/eval_subprocess_main.py) and its own process group: cancelling it is one signal to the group, and a crash cannot take the backend down. As a backstop, the subprocess also caps its own torch allocator at the admitted VRAM amount (AGENTCANVAS_VRAM_CAP_MB) — a job that badly exceeds its estimate crashes itself instead of its neighbours. This cap exists only for VRAM; RAM is checked at admission but not enforced afterwards (§6).

1.1 The smallest run

A graph name and an episode count; everything else defaults. The response is immediate — the run is queued, not started:

curl -X POST http://127.0.0.1:8000/api/eval/v2/start \
  -H "Content-Type: application/json" \
  -d '{"graph_name": "straightforward", "episode_count": 1}'
# → {"run_id": "20260704_200243", "status": "queued", "via_subprocess": true}

The response comes back immediately: the run is queued, not started. A second or two later the scheduler admits it — this graph has run before, so the charge is what was measured last time, not what any profile declared. The subprocess runs its episode, writes summary.json and a _DONE marker, and is reaped — leaving one more layer of measurements in the calibration file.

2. Lifecycle of one job

A job moves through four states, and every transition is written to disk as it happens — restart the backend and the completed history is still there.

Four states, everything on disk. The two dashed exits are the only ways off the happy path.

• submit (job_scheduler.py · JobScheduler.submit) — reject impossible declarations on the spot; write spec.json + a pending summary; enqueue. The models the graph needs were already loaded by the API handler — so at admit time they cost the job almost nothing.
• admitted (JobScheduler._admit, §3) — picked FIFO when the gates clear.
• running (JobScheduler._spawn) — own process group, output teed to the run dir, the admitted VRAM amount passed in as the allocator cap; an observation window opens and tracks the job's memory peaks until it ends (resource_stats.py · note_job_started).
• reaped (JobScheduler._reap) — final status from the table below; model servers nobody uses any more are unloaded; the window sediments into the calibration file (resource_stats.py · note_job_finished).

3. The measured admission gate

Admission guards against two failures at once: starting a job that doesn't fit (an OOM can kill several runs), and holding back a job that would fit (an idle GPU). Every second, JobScheduler._admit walks the queue in submission order and applies four checks:

1. canvas lock — someone is using canvas Play; the GPU is theirs, nothing admits.
2. exclusive — a job marked exclusive_gpu waits for an empty machine, and blocks everything else while it runs.
3. the fit check — for each resource the machine can measure (VRAM, RAM): estimated need ≤ measured free − standing reservations. A job that doesn't fit is skipped, not blocking — smaller jobs behind it may start first.
4. fallback — a machine with no sampler at all (unit tests) falls back to the old declared bookkeeping: marginal_vram_mb ≤ usable − Σ declared.

The trust ladder. Admission takes the highest rung that has a number (JobScheduler._resolve_estimate); the rung is recorded on the job as admit_basis.

One race needs closing: a just-admitted job hasn't allocated its memory yet (models take a minute to load), so it holds a reservation — its admitted amount minus what it has actually been measured using (JobScheduler._pending_reservations). As the real allocation appears in the samples, the reservation decays to zero on its own:

4. Measurement, calibration, estimation

Everything measurement-related lives in one module, services/resource_stats.py. It has three jobs: work out whose memory each megabyte is (attribution, §4.1), remember what each graph and each model actually used (calibration, §4.2), and turn those records into the number the gate needs (estimation, §4.3).

4.1 Attribution — whose memory is this?

Two per-PID lists arrive every second — GPU memory from nvidia-smi pmon (resource_sampler.py · _gpu_processes; this variant also sees the EGL graphics contexts habitat and MatterSim render through) and RAM from one /proc walk (resource_stats.py · _proc_table). Ownership is decided by walking up each process's parent chain (resource_stats.py · _attribute_pid):

Whose memory is this? Walk up the parent chain until an owner is found.

1. shared model server — the process or an ancestor is a server the registry started;
2. running job — the chain reaches a job's subprocess (parent links survive workers moving into their own process groups);
3. external — everything else; charged to nobody, but the gate still feels it through "measured free".

4.2 Sedimentation — writing the calibration store

While a job runs, its window keeps the peak of everything it owns, per resource; on reap it is written into outputs/system/resource_calibration.json (running average + observed max). Two guards keep the data honest: a window that never got a sample writes nothing (a sampler outage must not record fake zeros), and a running server that used 0 MB is recorded as a real zero — "measured zero" ≠ "never measured". Entries are keyed to the nodeset's source hash (content_hash.py · hash_nodeset_tree); pre-RAM calibration files migrate automatically (resource_stats.py · _migrate_v1).

4.3 Estimation — GET /api/eval/v2/estimate

How the number is assembled — and what it must fit into.

Each block states its origin (resource_stats.py · estimate_tree_mb / estimate_shared_mb): measured (this worker count was seen before), fitted (a line through ≥2 worker counts), scaled (one observation, scaled), or hint — the author preset expected_vram_mb / expected_ram_mb (bases.py · BaseNodeSet), used when nothing was measured; presets survive code edits because a model's footprint is its weights, and a real measurement replaces them once one exists. One honesty rule: if any component has no number, the total is null — a partial sum pretending to be a total is worse than no answer. Ask without running anything: GET /api/eval/v2/estimate (eval.py · estimate_eval_v2) returns the breakdown and max_workers.

5. Observability — one snapshot explains the decision

When a job sits in the queue and you want to know why, one call answers it: GET /api/eval/v2/queue (JobScheduler.list_active). Below is a real snapshot, taken a few seconds after a job was admitted. Reading it: the job was charged 2049 MB of VRAM and 3515 MB of RAM, both measured numbers ("admit_basis": "calibrated"). The sampler has already watched 543 MB / 2480 MB of that actually materialize, so the job's outstanding reservations have decayed to 1506 / 1035:

{
  "running": [{"run_id": "20260705_140954",
               "admit_charges": {"vram": 2049, "ram": 3515},
               "admit_basis": "calibrated", "marginal_vram_mb": 0}],
  "queued": [],
  "measured_free_mb": {"vram": 23811, "ram": 46346},
  "pending_reservation_mb": {"vram": 1506, "ram": 1035},
  "attribution": {"resources": {
    "vram": {"used_total_mb": 765, "attributed_mb": 543, "external_mb": 222,
             "shared_mb": {}, "jobs_mb": {"20260705_140954": 543}},
    "ram":  {"used_total_mb": 13540, "attributed_mb": 4154, "external_mb": 9386,
             "shared_mb": {"policy_adapter_vlnce": 1674},
             "jobs_mb": {"20260705_140954": 2480}}}}
}

Two older fields, usable_vram_mb / reserved_vram_mb (not shown above), still exist for the no-sampler fallback. The fields a decision actually uses are measured_free_mb, pending_reservation_mb, and each job's admit_charges + admit_basis. To cancel a run: POST /runs/{run_id}/cancel — a queued job is cancelled on the spot; a running one gets a SIGTERM to its whole process group.

6. Boundaries — trade-offs, scope cuts, physics

Nothing in this table is an accident discovered after the fact. Each row is a decision with its reasoning attached: a trade-off (the alternative was considered and rejected), a scope cut (deliberately not built yet, with the revisit trigger named), or physics (what a once-per-second sampler simply cannot see). They are written down so nobody has to rediscover them the hard way.