ADR-server-004 — AgentCanvas

Date: 2026-06-15
Status: accepted
Field: server

Context

Server mode (ADR-server-001) cuts a process boundary through the middle of a graph: some nodesets run in their own subprocess (env isolation, e.g. Habitat on Py3.8; and the reserved cross-machine capability), and the framework talks to them over HTTP. Three things that are free inside one process break at that boundary — any Python object passed by reference, data handed node-to-node in memory, and a State Container shared by all nodes — but they share one root: the single-process shared-memory assumption is severed. The wire codec was the most broken facet: serialize_value → _make_json_safe (serialization.py) only converted ndarray / numpy scalars / dict / list; a torch.Tensor, PIL.Image, bytes, set, or custom object on an ANY port reached the JSON encoder and 500'd (TODO #67). The same graph ran fine local (by-reference) but crashed server — a position-dependence bug. Two adjacent gaps shared the root: subprocess logs/errors had no return path (a handler exception became HTTPException 500 → the proxy demoted it to a swallowed {"error": ...} value the canvas never saw — TODO #54), and a shared server owning a mutable state container (a race under worker_count>1) had no guardrail (TODO #68). A prior prototype (0fd2df90) had landed cross-process container access (broker + RemoteContainerProxy) but on JSON, synchronous, single-worker. The contract for "what may cross a server boundary, and how" was undefined. This ADR is the formalization of that GAPS-flow design.

Decision

Adopt one principle — only passive data crosses the process boundary — and realize it as three moves plus one explicit non-goal, all under the invariant of position transparency (the graph must behave identically whether a nodeset runs local / local-server / remote-server).

Move 1 — msgpack data codec (transport, #67). pack_body / unpack_body (serialization.py) encode the whole {"inputs", "config"} body in one shot. A single blob ExtType — framed [tag:u8][meta_len:u32][meta(msgpack)][raw bytes] — carries the binary types JSON can't hold (ndarray, torch.Tensor, PIL.Image) as raw bytes (no base64); bytes ride msgpack's native bin type. Decoding is type-driven with receiver-side degrade: a torch/PIL blob decoded in an env lacking that type comes back as ndarray rather than crashing (cross-boundary = cross-env; rebuilding a wrapper type is not free across envs). /call negotiates by Content-Type and still accepts the legacy JSON encoding during a migration window. The container reverse-channel uses the same codec.
Move 2 — state stays home, reachable cross-process (state, #17/#68). Mutable state has exactly two legal homes: the executor's graph_state / home container, or a replicated server subprocess's per-worker private copy (a shared server must be stateless). Other processes reach a container transparently through the executor broker + RemoteContainerProxy (ctx.containers["x"].read/write unchanged) — the 0fd2df90 prototype, normalized: broker forwarding async, payload msgpack, executor URL from Settings (resolve_executor_url), and a load-time #68 guardrail (registry._check_container_ownership, warn) against a shared server owning state.
Move 3 — reverse channel for logs/errors (observability, #54). Subprocesses POST structured events to the executor's POST /api/internal/events (internal_events.py), which republishes them on the ErrorBus → error_event WebSocket frame, so server-node logs/errors surface on the canvas like local ones. Handler exceptions are first-classed (node/execution scope + traceback) instead of swallowed.
Non-goal — no co-location transport optimization. All wire data round-trips through the executor even when producer and consumer share a subprocess. Wire data is light (≤ a few MB); the heavy state already stays home (Move 2); and short-circuiting the executor would make the values invisible to the canvas — directly against AgentCanvas's "see the data flow" premise.

Alternatives

(a) Document + enforce "ANY across a server boundary = JSON-safe-after-numpy; opaque objects stay subprocess-local" (#67 option a). Punts the 500 onto the node author. Kept only as the pre-landing stopgap, not the fix.

(b) Extend _make_json_safe with a JSON codec table (torch/PIL → base64) (#67 option b, JSON variant). Still base64 (+33%) and still text-parse on a big array. msgpack subsumes it with raw bytes and a cleaner ExtType, so the codec-table idea was kept but moved onto msgpack.

(c) A handle/pointer wire type for objects that must stay process-local (#67 option c). Rejected. A handle valid only in one process re-introduces position-dependence — the opposite of ADR-server-001's "deployment topology is a config decision". Live objects already stay home by ownership (a nodeset holds its simulator/model/planner and only emits passive outputs); no wire type is needed to express that.

pickle / cloudpickle for cross-process objects. Rejected — cross-boundary = cross-env (different Python / library versions); only version-neutral, self-describing bytes are safe.

Co-located data relay (skip the executor for same-subprocess edges). Rejected as the non-goal above: conflicts with observability and the savings are marginal once heavy state stays home.

WebSocket long-connection for log push. Rejected for v1 — the executor would hold N persistent connections + reconnection logic; a POST endpoint mirrors the already-working container reverse channel with no long-lived state.

Rationale

One boundary deserves one contract. Reframing the problem from "how do we ship objects across processes" to "what may cross" (only passive data) collapses the whole transport question into "how do we encode bytes" — and msgpack answers that with version-neutral, self-describing, raw-binary encoding plus a clean degrade story for types a receiving env lacks. Routing log/error events through the existing /api/internal reverse channel into the ErrorBus reuses infrastructure instead of inventing a parallel one. Position transparency is the invariant every move preserves: the proxy node always exposes forward(inputs) → outputs and the graph never learns where a nodeset runs. The non-goal protects clean, observable execution over a marginal transport saving.

Verified. 63 backend unit tests (codec round-trip incl. torch-degrade, /call + /containers Content-Type negotiation, #68 guardrail, URL resolver, event push); a real auto-host subprocess E2E (msgpack /call + JSON migration window + container state persistence); and a 10-episode / 10-worker explore-eqa run (SR 0.4 ≈ 0.42 verified baseline, zero 500s, all 21 fresh-spawned servers across shared + replicated nodesets).

Residual (surfaced, not hidden): cross-nodeset access to a replicated nodeset's containers under worker_count>1 is not yet routable — the home registry resolves an untagged URL and the broker can't disambiguate N executors in the eval topology. Emitted as a load-time warning rather than a silent 404; tracked under TODO #17. Owner-local container access (the common case, e.g. explore_eqa_tsdf) is unaffected.

Affected docs

core/decisions/server/index.html — add ADR-server-004 row
core/decisions/index.html — bump server count + total
design-docs/operations/plugin-servers.html — §7.1 rewritten to msgpack; new §7.3 reverse log/error channel; changelog
core/roadmap.html — #67 (✅) / #68 (✅) / #54 / #17 update notes