AgentCanvas / Pages / Developer Guide / Capabilities / Isolated Runtime Environments
2026-10-17

Habitat-Sim wants Python 3.8; modern HF models want torch โ‰ฅ 2.x; the backend runs Python 3.10+. They cannot share an interpreter โ€” so any BaseNodeSet can opt into server mode: the framework spawns it as its own subprocess on its own interpreter (server_python), auto-generates the HTTP surface from the same PortDefs the nodeset already declares, and registers proxy nodes that look identical on the canvas. Zero server code per nodeset; the researcher cannot tell which mode is active. Under batch eval the same mechanism fans out to N tagged subprocesses, one leased per episode (ADR-eval-002).

backend process agentcanvas env ยท py3.10+ isolated subprocess server_python ยท e.g. py3.8 + habitat GraphExecutor proxy node same node_type ยท forward() โ†’ HTTP AutoServerApp nodeset + env panel tool.forward() ยท sim state load: /manifest โ†’ proxies POST /call/{fn} /env-panel/{method} server_python picks the interpreter per nodeset โ€” py3.8 Habitat and the py3.10 backend never share a process batch eval: ร—N tagged subprocesses per env nodeset (#0โ€ฆ#N-1), leased one per episode (worker_count=N)

Design docs: Server Mode ยท MCP Projection ยท Env Panels


1. Problem

AgentCanvas runs in agentcanvas (Python 3.10+, modern torch / HF / FastAPI โ€” the primary env, ADR-platform-004). Research tools it drives often need different environments:

These cannot coexist in one interpreter. AgentCanvas solves this by running each nodeset in its own subprocess with its own interpreter โ€” transparently to the canvas.

2. Two Loading Modes

Mode Command Where tools run Use case
Local POST /api/components/nodesets/{name}/load In the backend process Same-env tools, fast I/O
Server POST /api/components/nodesets/{name}/load?mode=server In a separate subprocess Different Python env, GPU isolation, parallel workers

In both modes the tools appear as identical canvas nodes โ€” the researcher doesn't need to know which mode is active. Under batch eval, worker_count > 1 auto-routes env nodesets to server mode even if the graph was loaded locally (registry.py ยท load_nodeset, gated on parallelism="replicated").

3. Auto-Server Architecture

When mode=server, the framework performs these steps automatically:

1. WorkspaceComponentRegistry._load_nodeset_as_server(name, nodeset, worker_count=N)
2. Read server_python from nodeset class (or default to sys.executable)
3. Allocate N free TCP ports (one subprocess per worker)
4. Launch subprocess(es):
     PYTHONPATH=... {server_python} -m app.server.auto_host \
       --file {source_file} --class {class_name} --port {port}
5. Subprocess starts AutoServerApp:
     a. Instantiate nodeset env panel (if any) for the BaseNodeSet class
     b. Call on_startup() โ†’ instantiate nodeset, await initialize()
     c. Introspect get_tools() โ†’ read input_ports / output_ports
     d. Auto-generate ServerFunction per tool
     e. Serve HTTP: /manifest + /call/{fn} + /env-panel/* + /health
6. Framework fetches /manifest from each subprocess
7. Generate proxy canvas nodes from manifest
8. Register proxy nodes in the node catalog (once; proxies route to the right subprocess at call time)

3.1 AutoServerApp Introspection

AutoServerApp reads each tool's PortDef declarations and converts them to ServerFunction entries:

# From auto_server_app.py โ€” get_functions()
for tool in self._nodeset.get_tools():
    functions.append(ServerFunction(
        name=tool.node_type,
        description=tool.description,
        input_ports=_portdefs_to_schemas(tool.input_ports),
        output_ports=_portdefs_to_schemas(tool.output_ports),
        config_schema=tool.config_schema,
        handler=_make_handler(tool, self._get_inference_lock, self._containers),  # serialized=True nodes fire under the one lock
    ))

No manual port re-declaration needed โ€” the same PortDef used for local mode drives the server API. The manifest.FunctionSchema.name is the tool's node_type, so local and server-mode proxies register under the same node_type (TODO #33 closed).

3.2 MCP Projection (/mcp)

Full design: MCP Projection.

The same introspected manifest has a second face. When the mcp SDK is importable in the server's environment, ServerApp._build_app additionally mounts a native MCP endpoint at /mcp (Streamable HTTP), so any MCP client โ€” Claude Code, codex โ€” can call the nodeset's tools directly. Every manifest function becomes an MCP tool with zero per-tool code: mcp_projection.py synthesizes each tool's inputSchema from the same declarations (wire_type โ†’ JSON type, optional=False โ†’ required, config_fields โ†’ enum / bounded-number / boolean properties), and tools/call invokes the registered handler in-process with the /call route's JSON-regime serialization. IMAGE output ports come back as MCP ImageContent (base64 PNG); other ndarray outputs are compacted to shape/dtype stubs so a depth map never floods the model's context.

Session policy follows the nodeset's statefulness. BaseNodeSet.mcp_exclusive (ClassVar, default None) derives from the parallelism contract: "shared" nodesets (stateless model tools) serve concurrent MCP clients through a stateless transport, while "replicated" nodesets (stateful envs โ€” one simulator per process) allow at most one live MCP session โ€” a second initialize is rejected with 409 and a readable JSON-RPC error until the first client disconnects (its DELETE, or a 600 s idle timeout). The --mcp-tools flag narrows the MCP tool list (e.g. keep reset/evaluate away from an agent); /manifest and /call are never affected by it. On py3.8 hosts (ac-vlnce) the SDK import fails and the server runs manifest-only, unchanged.

# any nodeset server now doubles as an MCP server (needs the mcp SDK in its env)
python -m app.server.auto_host --file workspace/nodesets/model/model_sam.py \
    --class SamNodeSet --port 9200 --mcp-tools segment_auto

# one-line client config (Claude Code)
claude mcp add --transport http sam http://127.0.0.1:9200/mcp

The projection is additive โ€” no node class changes, and the manifest protocol behaves identically with or without it (app/server/test_mcp_projection.py). Two practical notes: pin mcp==1.27.0 in nodeset envs (mcp 2.0 changed the lowlevel API; under it the server degrades to manifest-only), and mind output-port payloads โ€” ports that carry graph-oriented bulk (e.g. model_sam__segment_points' always-on embedding envelope, โ‰ˆ5.6 MB base64) will overflow an MCP client's tool-output budget, so pick context-sized tools for the allowlist (segment_auto's envelope is KB-scale).

4. Proxy Nodes

After fetching the manifest, the framework generates proxy canvas nodes โ€” BaseCanvasNode subclasses that forward forward() calls to the remote server via HTTP.

Canvas node forward()  โ†’  HTTP POST /call/{fn}  โ†’  subprocess: tool.forward()  โ†’  HTTP response  โ†’  return dict

Proxy nodes have the same node_type, input_ports, output_ports, and category as the original tool. They're registered in the node catalog exactly like local nodes, so the GraphExecutor treats them identically.

At call time, a proxy checks the running executor for a per-worker server_url_overrides entry (ADR-eval-002 PB-1.5, proxy.py::_make_forward); if present, the request is routed to that worker's URL, otherwise it falls back to the globally-registered URL. The call itself goes through AutoHostClient (server/client.py) โ€” the one HTTP client every framework-side caller of an auto_host uses โ€” and a server-side error (CallError) or an unreachable server (ServerUnreachable) raises out of the proxy's forward(), so the firing is convicted like a local node's exception.

5. Env Panel Bridging

Env nodesets own mutable episode state (current split, current episode, last observation) that must live inside the subprocess where the simulator actually runs. ADR-server-002 formalises this with a generic BaseEnvPanel contract:

This replaced the earlier pattern of smuggling episode control through hidden env_*__ep_* tool nodes. 5 hidden tools + api/canvas/env.py (233 lines) + app/env_controller.py (343 lines) were deleted; episode management is fully decoupled from the tool-call path.

6. Worker-Pool Fan-Out (ADR-eval-002)

For batch evaluation, worker_count > 1 extends server mode to N parallel isolated subprocesses per env nodeset:

ensure_nodesets_for_graph(graph, worker_count=N)
  โ”œโ”€ for env nodeset in graph:
  โ”‚    โ””โ”€ _load_nodeset_as_server(name, ns, worker_count=N)
  โ”‚         โ”œโ”€ allocate N free ports
  โ”‚         โ”œโ”€ spawn N auto_host subprocesses
  โ”‚         โ”œโ”€ fetch manifest from first; generate proxy nodes once
  โ”‚         โ””โ”€ register N tagged RemoteEnvPanelProxy instances: "{name}#0"โ€ฆ"{name}#{N-1}"
  โ””โ”€ non-env nodesets stay singleton (one subprocess, shared across workers)

The Fleet (components/fleet.py) then gives each worker one WorkerHandle; each handle carries:

At worker_count=1 both overrides are empty and the code path collapses to the singleton case (zero behaviour change). On eval shutdown, all N tagged subprocesses are unloaded together with rollback if any failed to come up.

7. Serialized calls (serialized)

Some nodes (e.g. policy_adapter_vla__predict) must not run two forwards on their device at once while K parallel workers call them through one parallelism="shared" subprocess. The hosting AutoServerApp holds one asyncio.Lock for that:

This replaced the ADR-eval-002 PC batched-inference rendezvous (BatchedInferenceServer, batched / batch_dim) on 2026-10-17: its one consumer unrolled every batch sequentially, so the lock was all it provided.

8. How to Use

8.1 Specifying a Different Interpreter

Add server_python to the nodeset class:

class SlamNodeSet(BaseNodeSet):
    name = "slam"
    description = "SLAM spatial tools"
    server_python = "/opt/ros/noetic/bin/python3"

    def get_tools(self) -> list:
        return [LocalizeTool(), MapQueryTool(), FrontierTool()]

For the ac-vlnce env:

class EnvHabitatNodeSet(BaseNodeSet):
    name = "env_habitat"
    server_python = "python"
    env_panel = HabitatEnvPanel  # ADR-server-002 โ€” lives in the subprocess

8.2 Loading via API

# Load in local mode (default)
curl -X POST http://localhost:8000/api/components/nodesets/sam/load

# Load in server mode (auto-hosted subprocess)
curl -X POST http://localhost:8000/api/components/nodesets/slam/load?mode=server

# Unload (works for all modes, including worker-pool)
curl -X POST http://localhost:8000/api/components/nodesets/slam/unload

Batch eval loads nodesets via ensure_nodesets_for_graph(graph, worker_count=N) โ€” there is no public ?worker_count=N query parameter; fan-out is driven by EvalConfig.worker_count.

8.3 Opting Into Serialized Calls

class MyPolicyForward(BaseCanvasNode):
    node_type = "my_policy__forward"
    serialized = True  # one forward in flight on this subprocess's device
    input_ports = [
        PortDef("raw_obs", WIRE_OBSERVATION),
        PortDef("hidden_in", WIRE_ANY),   # explicit per-caller state
        ...
    ]
    output_ports = [
        PortDef("action", WIRE_ACTION),
        PortDef("hidden_out", WIRE_ANY),  # explicit per-caller state
    ]

9. Key Files

File Role
agentcanvas/backend/app/server/auto_server_app.py AutoServerApp โ€” introspects nodeset, generates ServerFunction entries, holds the one inference lock for serialized nodes, instantiates nodeset env panel in-process
agentcanvas/backend/app/server/auto_host.py CLI entry point: python -m app.server.auto_host --file ... --class ... --port ...
agentcanvas/backend/app/server/server_app.py ServerApp base โ€” HTTP endpoints, manifest serving, /env-panel/* routes (ADR-server-002)
agentcanvas/backend/app/server/base_server.py BaseServer โ€” subprocess lifecycle, health check, manifest fetch
agentcanvas/backend/app/server/client.py AutoHostClient โ€” the one framework-side HTTP client for an auto_host (call, panel, manifest, health, evict, container ops)
agentcanvas/backend/app/server/invoke.py invoke โ€” the one server-side function call behind POST /call and MCP tools/call
agentcanvas/backend/app/server/proxy.py generate_proxy_nodes() + _make_forward โ€” proxy BaseCanvasNode classes with per-worker URL routing
agentcanvas/backend/app/server/manifest.py ServerManifest, FunctionSchema, PortSchema โ€” manifest data model
agentcanvas/backend/app/server/serialization.py Wire-type (de)serialization for HTTP transport โ€” base64 images, numpy arrays, nested dicts
agentcanvas/backend/app/server/mcp_projection.py Native MCP projection โ€” manifest โ†’ MCP tool schemas, in-process tools/call, mcp_exclusive session gate (ยง3.2)
agentcanvas/backend/app/components/registry.py _load_nodeset_as_server(worker_count=N), tagged-proxy registration, rollback on partial failure
agentcanvas/backend/app/components/env_panel.py BaseEnvPanel ABC + RemoteEnvPanelProxy (ADR-server-002)
agentcanvas/backend/app/components/launcher.py vehicle_for + Launcher.build โ€” local / native / container routing and server assembly
agentcanvas/backend/app/components/fleet.py Fleet + WorkerHandle โ€” one handle per worker over the tagged copies, for batch eval

Status

Item Status Notes
Local loading mode Done load_nodeset(name, mode="local") โ€” in-process
Server loading mode Done load_nodeset(name, mode="server") โ€” subprocess with auto-server
AutoServerApp (introspect PortDef โ†’ ServerFunction) Done Zero manual port re-declaration
auto_host CLI (python -m app.server.auto_host) Done --file, --class, --port args
BaseServer (subprocess lifecycle, health check, manifest) Done start/stop/restart, _wait_for_health, fetch_manifest
Proxy node generation Done generate_proxy_nodes() + per-worker URL routing in _make_forward
server_python ClassVar Done Selects interpreter for subprocess
Load/unload API endpoints Done POST /api/components/nodesets/{name}/load?mode=server
Manifest protocol (/manifest endpoint) Done ServerApp serves manifest with FunctionSchema entries; node_type is the schema name
MCP projection (/mcp endpoint) Done Native Streamable HTTP MCP face of the manifest; --mcp-tools allowlist, mcp_exclusive session gate; py3.8 hosts degrade to manifest-only
Env panel bridging (ADR-server-002) Done /env-panel/* routes, RemoteEnvPanelProxy, env panel lives in subprocess
Worker-pool fan-out (ADR-eval-002 PB) Done worker_count > 1 spawns N tagged subprocesses + N tagged proxies with rollback
Per-worker URL routing (ADR-eval-002 PB-1.5) Done server_url_overrides consulted by proxy._make_forward
Serialized calls (replaced the ADR-eval-002 PC batched tier, 2026-10-17) Done serialized opt-in on BaseCanvasNode; one lock per AutoServerApp
Docker server mode Planned Feature TODO F7 โ€” unblocked by ADR-eval-002; subprocess spawn is the only seam to swap
AgentCanvas docs