3Isolated Runtime Environments

Research tools often require conflicting Python environments. Any BaseNodeSet can run in server mode in its own interpreter — the framework auto-generates an HTTP server from the tool's port definitions. Zero extra server code. Under batch eval, the same mechanism fans out to N tagged subprocesses for per-worker isolation (ADR-eval-002).

Design docs: Server Mode · Env Panels

1. Problem

AgentCanvas runs in agentcanvas (Python 3.10+, modern torch / HF / FastAPI — the primary env, ADR-platform-004). Research tools it drives often need different environments:

• Habitat-Sim 0.1.7 — pinned to Python 3.8 (ac-vlnce env) by its binary wheel
• SAM / modern HF models — need torch ≥ 2.x, won't load under old CUDA
• SLAM / ROS — Python 3.8 with ROS Noetic bindings

These cannot coexist in one interpreter. AgentCanvas solves this by running each nodeset in its own subprocess with its own interpreter — transparently to the canvas.

2. Two Loading Modes

In both modes the tools appear as identical canvas nodes — the researcher doesn't need to know which mode is active. Under batch eval, worker_count > 1 auto-routes env nodesets to server mode even if the graph was loaded locally (registry.py · load_nodeset, gated on parallelism="replicated").

3. Auto-Server Architecture

When mode=server, the framework performs these steps automatically:

1. WorkspaceComponentRegistry._load_nodeset_as_server(name, nodeset, worker_count=N)
2. Read server_python from nodeset class (or default to sys.executable)
3. Allocate N free TCP ports (one subprocess per worker)
4. Launch subprocess(es):
     PYTHONPATH=... {server_python} -m app.server.auto_host \
       --file {source_file} --class {class_name} --port {port}
5. Subprocess starts AutoServerApp:
     a. Instantiate nodeset env panel (if any) for the BaseNodeSet class
     b. Call on_startup() → instantiate nodeset, await initialize()
     c. Introspect get_tools() → read input_ports / output_ports
     d. Auto-generate ServerFunction per tool
     e. Serve HTTP: /manifest + /call/{fn} + /env-panel/* + /health
6. Framework fetches /manifest from each subprocess
7. Generate proxy canvas nodes from manifest
8. Register proxy nodes in NODE_HANDLERS (once; proxies route to the right subprocess at call time)

3.1 AutoServerApp Introspection

AutoServerApp reads each tool's PortDef declarations and converts them to ServerFunction entries:

# From auto_server_app.py — get_functions()
for tool in self._nodeset.get_tools():
    functions.append(ServerFunction(
        name=tool.node_type,
        description=tool.description,
        input_ports=_portdefs_to_schemas(tool.input_ports),
        output_ports=_portdefs_to_schemas(tool.output_ports),
        config_schema=tool.config_schema,
        handler=_make_handler(tool, self._batched_server),  # batched server passed for batched=True nodes
    ))

No manual port re-declaration needed — the same PortDef used for local mode drives the server API. The manifest.FunctionSchema.name is the tool's node_type, so local and server-mode proxies register under the same node_type (TODO #33 closed).

4. Proxy Nodes

After fetching the manifest, the framework generates proxy canvas nodes — BaseCanvasNode subclasses that forward forward() calls to the remote server via HTTP.

Canvas node forward()  →  HTTP POST /call/{fn}  →  subprocess: tool.forward()  →  HTTP response  →  return dict

Proxy nodes have the same node_type, input_ports, output_ports, and category as the original tool. They're registered in NODE_HANDLERS exactly like local nodes, so the GraphExecutor treats them identically.

At call time, a proxy checks the running executor for a per-worker server_url_overrides entry (ADR-eval-002 PB-1.5, proxy.py::_make_forward); if present, the request is routed to that worker's URL, otherwise it falls back to the globally-registered URL.

5. Env Panel Bridging

Env nodesets own mutable episode state (current split, current episode, last observation) that must live inside the subprocess where the simulator actually runs. ADR-server-002 formalises this with a generic BaseEnvPanel contract:

• BaseNodeSet.env_panel: type[BaseEnvPanel] | None — opt-in ClassVar on the nodeset class
• AutoServerApp.on_startup() instantiates the env panel inside the subprocess with _context = {"mode": "local", "server_url": None, "nodeset_name": ...} — the env panel is local to the subprocess, not the framework
• ServerApp._build_app() mounts generic /env-panel/info, /env-panel/{method} routes that proxy method calls to the subprocess env panel
• Framework side: RemoteEnvPanelProxy (one per worker) is registered in the global env panel registry. Calls go HTTP → subprocess env panel → live simulator state

This replaced the earlier pattern of smuggling episode control through hidden env_*__ep_* tool nodes. 5 hidden tools + api/canvas/env.py (233 lines) + app/env_controller.py (343 lines) were deleted; episode management is fully decoupled from the tool-call path.

6. Worker-Pool Fan-Out (ADR-eval-002)

For batch evaluation, worker_count > 1 extends server mode to N parallel isolated subprocesses per env nodeset:

ensure_nodesets_for_graph(graph, worker_count=N)
  ├─ for env nodeset in graph:
  │    └─ _load_nodeset_as_server(name, ns, worker_count=N)
  │         ├─ allocate N free ports
  │         ├─ spawn N auto_host subprocesses
  │         ├─ fetch manifest from first; generate proxy nodes once
  │         └─ register N tagged RemoteEnvPanelProxy instances: "{name}#0"…"{name}#{N-1}"
  └─ non-env nodesets stay singleton (one subprocess, shared across workers)

EnvWorkerPool then leases one WorkerHandle per episode; each handle carries:

• env_panel_overrides {nodeset_name: tagged_proxy} — consulted by executor.get_env_panel(name) before the global registry, so set_episode/reset calls hit the right subprocess
• server_url_overrides {nodeset_name: tagged_url} — consulted by executor.get_server_url(name), so in-graph tool calls from the proxy node hit the same subprocess

At worker_count=1 both overrides are empty and the code path collapses to the singleton case (zero behaviour change). On eval shutdown, all N tagged subprocesses are unloaded together with rollback if any failed to come up.

7. Batched Inference (ADR-eval-002)

Some nodes (e.g. policy_adapter_vla__predict) amortise better when K parallel callers share one GPU forward pass. BatchedInferenceServer is a per-AutoServerApp rendezvous tier:

• Opt-in on the node class: batched: ClassVar[bool] = True, batch_dim: ClassVar[str] = "<input_port_name>" (validated at scan time)
• AutoServerApp._make_handler routes batched=True calls through BatchedInferenceServer.submit(function_name, inputs, config)
• The server holds one _BatchQueue per (function_name, config_hash) — collects K submissions, calls the underlying handler once with stacked inputs (via SAMPLES_KEY/OUTPUTS_KEY markers), scatters the K result slices to K awaiting futures
• Restart-on-each-submit flush timer at flush_timeout_ms (default 50 ms) — flush on either batch_size == worker_count or timeout, whichever fires first
• Pure-functional contract: any per-call state must travel as explicit hidden_in / hidden_out ports on the node — the server never carries caller-scoped state between calls

This sits inside a single subprocess (one AutoServerApp instance), so it composes cleanly with the worker-pool fan-out: K parallel workers submit into the same batched queue hosted by the inference subprocess.

8. How to Use

8.1 Specifying a Different Interpreter

Add server_python to the nodeset class:

class SlamNodeSet(BaseNodeSet):
    name = "slam"
    description = "SLAM spatial tools"
    server_python = "/opt/ros/noetic/bin/python3"

    def get_tools(self) -> list:
        return [LocalizeTool(), MapQueryTool(), FrontierTool()]

For the ac-vlnce env:

class EnvHabitatNodeSet(BaseNodeSet):
    name = "env_habitat"
    server_python = "python"
    env_panel = HabitatEnvPanel  # ADR-server-002 — lives in the subprocess

8.2 Loading via API

# Load in local mode (default)
curl -X POST http://localhost:8000/api/components/nodesets/sam/load

# Load in server mode (auto-hosted subprocess)
curl -X POST http://localhost:8000/api/components/nodesets/slam/load?mode=server

# Unload (works for all modes, including worker-pool)
curl -X POST http://localhost:8000/api/components/nodesets/slam/unload

Batch eval loads nodesets via ensure_nodesets_for_graph(graph, worker_count=N) — there is no public ?worker_count=N query parameter; fan-out is driven by EvalConfig.worker_count.

8.3 Opting Into Batched Inference

class MyPolicyForward(BaseCanvasNode):
    node_type = "my_policy__forward"
    batched = True
    batch_dim = "raw_obs"  # must name an actual input port
    input_ports = [
        PortDef("raw_obs", WIRE_OBSERVATION),
        PortDef("hidden_in", WIRE_ANY),   # explicit per-caller state
        ...
    ]
    output_ports = [
        PortDef("action", WIRE_ACTION),
        PortDef("hidden_out", WIRE_ANY),  # explicit per-caller state
    ]

9. Key Files

Status