Reload & Code Freshness

The promise behind running agent-edited code: a run uses the code you meant — not a stale copy, not a half-updated mix. This page is the developer-facing internals of how that promise is kept (grounded in file·function), and the few places where today's code does not yet match the intended model. For how a run is launched and observed see Batch Eval System; for the parallelism contract that decides which servers are shared see ADR-server-003.

1. The mental model

Four invariants — everything else follows. The first three describe how code reaches a run; the fourth is the intended reload policy the code is still converging on (§4 marks where it doesn't yet hold).

• Disk is the source of truth — the frozen workspace/ plus an optional active overlay; nothing else is authoritative.
• A reload is a replace, never an in-place mutation — re-execute the file, swap in the new class.
• A run is an immutable snapshot — code is pinned when the run starts; live workers are never updated mid-run.
• Reload is manual-only (intended) — code refreshes when a human clicks Reload or calls the API, never automatically; a file change should only signal a reload, never perform one.

The smallest case. Edit one nodeset file, then watch two consumers diverge:

# edited on disk: workspace/nodesets/method/foo/foo.py

POST /api/eval/v2/start { "graph_name": "foo_graph", "episode_count": 1 }
#   → runs the NEW foo.py — the eval is a cold subprocess that re-scans disk at boot

# the live canvas still holds the OLD foo.py until you ask:
POST /api/components/reload
#   → shutdown_all → scan_all → initialize_all   (full re-discovery)

Two consumers, two freshness rules: the cold eval is fresh for free; the live backend waits for an explicit reload. The rest of the page is why, and the one consumer (shared servers) where even that isn't enough.

2. How the code does it

Re-import primitive. The registry never calls importlib.reload and never trusts the module cache. registry.py · _import_module pops any stale sys.modules entry and exec_modules the file from disk under a fresh synthetic name carrying a scan-cycle counter (mod_name = f"_ws_{subdir}_{stem}_{scan_cycle}"), so the new class object replaces the old. This is what makes invariant 2 true.

Eval = a pinned snapshot. An eval run is a fresh interpreter — job_scheduler.py · _spawn does subprocess.Popen([sys.executable, "-m", "app.eval_subprocess_main", …]), not a fork — and eval_subprocess_main.py · _build_local_registry_for_run re-runs registry.scan_all() from disk at startup; the graph is re-read fresh per submit (eval.py · _load_graph_by_name). All worker_count=K workers are spawned from the one resolved source_file (registry.py · _load_nodeset_as_server), each gated on /health before episodes start (base_server.py · _wait_for_health). Once up, a worker never re-reads disk — so a mid-run edit cannot reach it; you get the new code on the next run. This realizes invariant 3.

Live backend = manual reload. The long-lived backend scans once at startup (main.py · lifespan) and then holds that code for the canvas and for shared servers. To refresh it, POST /api/components/reload runs shutdown_all → scan_all → initialize_all (api/platform/components.py · reload_components) — a complete re-discovery (all parallelisms, policies, hooks, and deletions, because scan_all clears first).

3. Active-workspace overlay

An optional overlay copy of the workspace takes precedence over the committed frozen one, so a search loop (the architect pipeline) can try edited code without dirtying frozen for sibling sessions. Two scopes: process-wide (ACTIVE_WORKSPACE_DIR at backend startup — affects everything, including the live canvas) or per-eval-request (--workspace=… — reaches only that one eval subprocess).

Resolution is overlay-first everywhere. registry.py · scan_all runs two passes — frozen, then overlay, same name wins; graph-by-name checks the overlay before frozen (eval.py · _load_graph_by_name); and the overlay is threaded into the cold eval subprocess via the ACTIVE_WORKSPACE_DIR env at spawn (job_scheduler.py · _spawn), so the subprocess's own two-pass scan loads overlay code. After that scan, each nodeset's resolved source_file already points at the overlay copy if it redefines that nodeset — and that is what determines how each parallelism is handled:

Why shared is the only special case. local and replicated nodesets are owned by the eval subprocess — it scans them overlay-first and spawns them itself, so the overlay is picked up for free via source_file. A shared nodeset is different: it is a single long-lived server in the parent backend, shared across sessions, that the eval merely connects to (shared_urls.json, computed in eval.py · start_eval_v2). The overlay cannot just re-point or reload it without disturbing other sessions or tearing down a GPU model.

Shared-overlay handling (TODO #60). So at admit, before the eval starts, the scheduler content-hashes the overlay's nodeset tree against frozen (job_scheduler.py · _prepare_ephemerals, via content_hash.py · hash_nodeset_tree / resolve_overlay_source). If they differ, it spawns a tagged ephemeral auto_host child from the overlay source, rewrites that URL into shared_urls.json so the run attaches to the overlay server, and tears it down by tag on reap — the frozen singleton stays alive for everyone else. If they are identical, no ephemeral is spawned and the run reuses the frozen singleton (saving VRAM). So the shared version a run sees is content-addressed and decided atomically before any worker attaches.

4. Where it deviates from the mental model

Three places where today's code does not match the invariants / the intended manual-only policy. Each is real, shipped behaviour — named here rather than hidden behind the clean model above.

5. Implementation map — which function does what

All paths under agentcanvas/backend/. Functions are named, not line-pinned — grep the symbol; line numbers drift, names don't.

5.1 Registry — discovery, import, load (app/components/registry.py)

5.2 Eval pipeline — submit, schedule, subprocess

5.3 Servers, manual reload, watchers, wiring