AgentCanvas / Pages / Developer Guide / Capabilities / Customizable Node System
2026-07-13

A canvas node is a Python class. Subclass BaseCanvasNode, declare what it is (node_type, category, icon), what it plugs into (PortDef inputs and outputs with wire types), what users can tweak (ui_config), and what it does (an async forward()) β€” then drop the file into workspace/. The WorkspaceComponentRegistry discovers it at startup or hot-reload, and the frontend's GenericBlockRenderer draws it from metadata alone: sidebar entry, ports, inline widgets β€” no frontend code, no manual registration. Nodesets (BaseNodeSet) bundle related nodes under one shared lifecycle, so one initialize() loads the model they all use. This page is the reference for that surface.

1 Β· you write 2 Β· the registry discovers 3 Β· the canvas renders class MyNode(BaseCanvasNode): node_type = "myNode" input_ports Β· output_ports ui_config = NodeUIConfig(…) async def forward(…): … one .py file dropped in workspace/ auto-discovery registry.scan_all() WorkspaceComponentRegistry import β†’ find subclasses β†’ register My Node text result log_list Β· metrics GenericBlockRenderer β€” no custom .tsx scan node-schemas hot reload: edit the .py β†’ POST /api/components/reload β†’ the canvas updates β€” no restart, no frontend build

1. BaseCanvasNode

Design docs: Canvas System Β· Loop Control (for kind="control" nodes)

Every canvas element β€” blocks, composites, controls β€” inherits from BaseCanvasNode. The class uses ClassVars for declarative metadata and an async forward() method for runtime logic.

from __future__ import annotations
from app.components.bases import BaseCanvasNode, PortDef

class MyNode(BaseCanvasNode):
    # Identity
    node_type = "myCustomNode"
    display_name = "My Node"
    description = "Does custom processing"
    category = "custom"        # sidebar group
    icon = "Sparkles"          # Lucide icon name

    # Kind: "block" | "composite" | "control"
    kind = "block"

    # Ports
    input_ports = [
        PortDef(name="text", wire_type="TEXT", description="Input text"),
    ]
    output_ports = [
        PortDef(name="result", wire_type="TEXT", description="Processed output"),
    ]

    # Config (JSON Schema for the properties panel)
    config_schema = {"temperature": {"type": "number", "default": 0.7}}
    default_config = {"temperature": 0.7}

    async def forward(self, inputs: dict, ctx: Any) -> dict:
        text = inputs["text"]
        result = process(text, self.config.get("temperature", 0.7))
        return {"result": result}

1.1 Lifecycle Hooks

async def initialize(self) -> None:
    """Called once before first execution. Load models, allocate GPU memory."""

async def shutdown(self) -> None:
    """Called on teardown. Release resources."""

1.2 Instance Attributes

Set by the executor at graph-build time:

Attribute Type Description
self.config dict Per-node config from graph JSON
self.node_id str Node ID from graph JSON
self._log_buffer list[dict] Voluntary inner log entries (per-firing). Written by _self_log(), read by log().

1.3 Voluntary Logging β€” _self_log() / log()

The executor automatically captures port I/O and timing (the exterior log layer). Nodes add interior detail β€” assembled LLM prompts, API responses, token counts, intermediate scores β€” by calling _self_log() inside forward(). The buffer is drained via log() after forward() returns (ADR-observability-003).

async def forward(self, inputs: dict, ctx: Any) -> dict:
    prompt = self._build_prompt(inputs)
    self._self_log("prompt", prompt)                       # record what we asked
    response = await llm_complete(prompt)
    self._self_log("tokens_used", response.usage.total_tokens)
    return {"text": response.text}

Override log() to filter or add computed summaries; the default returns the raw _self_log() buffer.

1.4 _resolve_ports(config) β€” Dynamic Ports

Some nodes derive their ports from per-instance config rather than class-level declarations β€” IterIn / IterOut (ADR-dataflow-003) and ImageViewerSink (ADR-components-007) are the canonical examples. Override the _resolve_ports classmethod to return (input_ports, output_ports) computed from config:

@classmethod
def _resolve_ports(cls, config: dict) -> tuple[list, list]:
    ports_cfg = config.get("ports", [])
    resolved = [PortDef(p["name"], p["wire_type"], optional=True) for p in ports_cfg]
    # Sink nodes: inputs only
    return (resolved, [])

Default behaviour (on BaseCanvasNode) returns class-level ports unchanged. The graph executor, load-time validator (validate_graph_connectivity), and the frontend's portResolution.ts all consult this method β€” single source of truth. Expose the config to users via a ConfigField(name="ports", field_type="port_list", …) in ui_config.

1.5 Batched Inference β€” batched / batch_dim

Opt-in ClassVars (ADR-eval-002 PC-1) that route a node's forward() through the per-AutoServerApp BatchedInferenceServer rendezvous. K parallel worker callers submit per-sample inputs; the underlying forward() is invoked once with inputs stacked along batch_dim; K result slices scatter back to K awaiting futures.

from typing import ClassVar

class PolicyForwardNode(BaseCanvasNode):
    node_type = "my_policy__forward"
    batched: ClassVar[bool] = True
    batch_dim: ClassVar[str] = "rgb"   # input port carrying the per-sample slot

    input_ports = [
        PortDef(name="rgb", wire_type="IMAGE"),
        PortDef(name="hidden_in", wire_type="ANY"),
    ]
    output_ports = [
        PortDef(name="action", wire_type="ACTION"),
        PortDef(name="hidden_out", wire_type="ANY"),
    ]

Contract: the server is pure-functional β€” any per-call state (RNN hidden state, etc.) must travel on the wire as explicit input/output ports. register_node() validates at scan time that batch_dim names an actual input port.

2. PortDef and Wiring

Design doc: Wire Types

PortDef declares an input or output port. The name becomes the React Flow Handle ID on the frontend and the key in inputs / return dict on the backend.

@dataclass
class PortDef:
    name: str             # port name (= Handle id)
    wire_type: str        # "IMAGE", "ACTION", "TEXT", "DEPTH", etc.
    description: str = ""
    optional: bool = False  # if False, node won't fire until this port has data

Wire types are checked at connection time β€” only matching types can be wired together. WIRE_TYPES contains 12 inner types: IMAGE, DEPTH, ACTION, DISCRETE_ACTION, CONTROL, POSE, TEXT, BOOL, METRICS, OBSERVATION, STEP_RESULT, ANY (POSE was renamed from STATE in ADR-dataflow-004; ANY is a first-class catch-all, not a boundary-only escape hatch).

Any inner type T may be wrapped as LIST[T] β€” a consumer-side modifier (ADR-dataflow-005). Producer ports stay scalar T; a consumer port declared LIST[T] always sees a list[T], because the executor wraps scalar inputs to length-1 lists and concatenates fan-in in edge declaration order at the port-binding seam. Used for multi-image LLM prompts, multi-LLM debate, and search-population fan-in. Nested LIST[LIST[T]] is not supported in v1.

3. Node UI Config

Design doc: Canvas System

NodeUIConfig controls how GenericBlockRenderer draws the node β€” no custom .tsx needed.

from app.components.bases import NodeUIConfig, ConfigField, DisplayField

class MyNode(BaseCanvasNode):
    ui_config = NodeUIConfig(
        color="amber",           # Tailwind color key ("" = auto from category)
        layout="block",          # "block" | "strip" | "viewer" | "imageGrid" | "note"
        width="",                # explicit width (e.g. "44px" for strip gates)
        min_width="",
        max_width="",
        min_height="",
        rounding="",             # CSS rounding class (e.g. "rounded-r-lg")
        config_fields=[
            ConfigField(
                name="temperature",
                field_type="slider",
                label="Temperature",
                default=0.7,
                min=0.0, max=2.0, step=0.1,
            ),
            ConfigField(
                name="model",
                field_type="select",
                label="Model",
                options=[
                    {"value": "gpt-4o", "label": "GPT-4o"},
                    {"value": "claude-sonnet", "label": "Claude Sonnet"},
                ],
            ),
        ],
        display_fields=[
            DisplayField(
                name="response",
                display_type="log_list",
                data_key="text",
                max_visible=20,
                accumulate=True,
            ),
        ],
    )

3.1 ConfigField Types

ConfigField declares user-editable inline widgets.

field_type Widget Extra fields
"label" Read-only text β€”
"slider" Range input min, max, step
"text" Single-line input placeholder
"select" Dropdown options: [{value, label}]
"toggle" Checkbox / switch β€”
"textarea" Multi-line input placeholder
"port_list" Editable port list Used by IterIn, IterOut, LLMCallNode, ImageViewerSink for configurable ports (ADR-dataflow-003, ADR-components-007)

3.2 DisplayField Types

DisplayField declares read-only runtime data widgets that render WebSocket viewer_data payloads keyed by data_key (ADR-components-006, ADR-observability-003). Set accumulate=True to append entries over time (bounded by max_visible); False to replace.

display_type Widget
"image_viewer" Base64-encoded image rendered as <img>
"log_list" Scrollable list of entries with step numbers (supports accumulate)
"metric_table" Key–value metrics table (SPL, SR, nDTW, …)
"text_viewer" Plain string display β€” scalar shows one block, array shows a scrollable stack with hairline dividers (ADR-components-008)

3.3 Layout Modes

layout When to use
"block" Default β€” standard rectangle with title, ports, and config/display fields
"strip" Narrow vertical gate (IterIn, IterOut, GraphIn, GraphOut) β€” ports only, no body
"viewer" Display-sink nodes (TextViewer, TextScroll, ActionLog, Metrics) β€” hides config_fields, renders display_fields full-width
"imageGrid" ImageViewerSink β€” grid of image panels derived from config.rows / config.cols / config.ports (ADR-components-007)
"note" Free-text annotation node (NoteLayout) β€” a canvas sticky-note with no ports or runtime behaviour

4. BaseNodeSet

Design docs: Server Mode Β· Env Panels

A nodeset bundles multiple BaseCanvasNode tools under shared initialization and lifecycle. Tools in a nodeset load/unload atomically.

from __future__ import annotations
from typing import ClassVar
from app.components.bases import BaseNodeSet

class SamNodeSet(BaseNodeSet):
    name = "sam"
    description = "Segment Anything Model tools"

    # Optional: run this nodeset in its own interpreter via AutoServerApp (ADR-server-001).
    # None = use sys.executable.
    server_python: ClassVar[str | None] = "/path/to/sam/python"

    # Optional: control-plane panel (ADR-server-002) β€” BaseEnvPanel subclass.
    # WorkspaceComponentRegistry instantiates and registers it on load. Used for
    # episode control, checkpoint selection, etc.
    env_panel: ClassVar[type | None] = None

    # Optional: per-episode timeout for batch eval (ADR-eval-002). The batch
    # runner clamps each episode at max_steps * default_per_step_budget_sec.
    # Habitat ~2.0, LLM-heavy MapGPT ~30.0, framework default 30.0.
    default_per_step_budget_sec: ClassVar[float] = 30.0

    def get_tools(self) -> list:
        return [SegmentTool(), EmbedTool()]

    async def initialize(self, **kwargs) -> None:
        # Load SAM model, allocate GPU memory
        ...

    async def shutdown(self) -> None:
        # Release GPU memory
        ...

    async def get_eval_metadata(self) -> dict:
        # Env nodesets override this to expose env_name, splits,
        # episode_counts, metrics, supports_set_episode, step_budget.
        # Non-env nodesets return {} (default).
        return {}

Each tool returned by get_tools() is itself a BaseCanvasNode subclass instance with node_type, input_ports, output_ports, and forward().

5. Auto-Discovery

The WorkspaceComponentRegistry scans workspace/ subdirectories at startup and on hot-reload.

workspace/
  nodesets/{env,method,policy,model,common,other}/ *.py
                         β†’ BaseNodeSet subclasses, grouped by role (ADR-platform-008;
                           server-mode is per-nodeset via server_python, not a directory)
  policies/        *.py  β†’ BaseCanvasNode subclasses with policy metadata
  nodes/           *.py  β†’ BaseCanvasNode subclasses (registered immediately)

5.1 Scan Flow

1. WorkspaceComponentRegistry.scan_all()
2. Pass 1 β€” frozen workspace (Settings.workspace_dir):
   For each subdirectory (nodesets/ with its role buckets, policies/, nodes/):
     a. Import every .py file (or package with __init__.py)
     b. Find concrete subclasses of the target base class
     c. Instantiate and register
3. Pass 2 β€” active workspace overlay (Settings.active_workspace_dir, optional):
   Same scan against the active dir. register_node()'s last-write-wins on
   NODE_HANDLERS gives Python-resource override; same for _discovered_nodesets.
   None / unset = no Pass 2 = bit-identical behaviour. (ADR-components-009)
4. NodeSets: discovered but NOT eagerly loaded at scan time. They load via
   (a) explicit load_nodeset(name) from the NodeSet Manager page, or
   (b) ensure_nodesets_for_graph(graph_def) β€” auto-loads every nodeset
       referenced by a graph when the graph is opened or a run is
       dispatched (ADR-components-003).
5. Nodes: registered immediately in NODE_HANDLERS
6. Policies: registered in NODE_HANDLERS + policy registry

Non-Python resources (graph JSON, .exp.yaml, hooks.json) are not scanned into a dict; they're looked up per-call via resolve_graph_path(name) / resolve_exp_yaml_path(name) on the registry (active-first then frozen fallthrough). hooks.json uses full-file override (no merge).

5.2 Hot Reload

POST /api/components/reload
  β†’ WorkspaceComponentRegistry.unregister_all()
  β†’ WorkspaceComponentRegistry.scan_all()
  β†’ invalidate_cache()  # frontend re-fetches node schemas

6. Frontend Rendering

Design doc: Canvas System

The frontend's GenericBlockRenderer renders any registered node automatically from its metadata. When scan_all() completes (or on reload), the backend serves node schemas via GET /api/components/node-schemas. The frontend uses this to:

  1. Build sidebar catalog β€” nodes grouped by category
  2. Render node blocks β€” ports, colors, inline config controls, live display widgets from ui_config
  3. Validate wiring β€” type-check connections using wire_type (including LIST[T] compatibility)

No custom .tsx component is needed unless the node requires specialized visualization beyond what GenericBlockRenderer provides.

7. Key Files

File Role
agentcanvas/backend/app/components/bases.py BaseCanvasNode, BaseNodeSet, PortDef, ConfigField, DisplayField, NodeUIConfig
agentcanvas/backend/app/components/registry.py WorkspaceComponentRegistry β€” scanning, registration, ensure_nodesets_for_graph, load/unload
agentcanvas/backend/app/agent_loop/builtin_nodes.py NODE_HANDLERS dict, register_node() (scan-time batched/batch_dim validation) β€” moved out of the retired graph_executor.py module
agentcanvas/backend/app/standard/wire_types.py Wire type constants, LIST[T] helpers, serialization
agentcanvas/backend/app/server/batched_inference.py BatchedInferenceServer β€” batched-inference rendezvous (ADR-eval-002)
agentcanvas/frontend/src/canvas/nodes/agentloop/inner/GenericBlockRenderer.tsx Universal node renderer
agentcanvas/frontend/src/canvas/portResolution.ts Frontend mirror of _resolve_ports β€” applies config.ports overrides

Status

Item Status Notes
BaseCanvasNode (all 14 ClassVars) Done node_type, display_name, description, category, icon, kind, input_ports, output_ports, children, config_schema, default_config, ui_config, batched, batch_dim
PortDef Done name, wire_type, description, optional
_resolve_ports(config) classmethod Done Instance-level port override (ADR-dataflow-003, ADR-components-007)
_self_log() / log() Done Voluntary interior log layer (ADR-observability-003)
Batched inference (batched + batch_dim) Done Scan-time validated; routed via BatchedInferenceServer (ADR-eval-002)
NodeUIConfig attrs Done color, layout, width, min_width, max_width, min_height, rounding, config_fields, display_fields
ConfigField (7 widget types) Done label, slider, text, select, toggle, textarea, port_list
DisplayField (4 display types) Done image_viewer, log_list, metric_table, text_viewer
Layout modes (5) Done block, strip, viewer, imageGrid, note
Wire types (12 inner + LIST[T]) Done POSE replaced STATE (ADR-dataflow-004); adds DISCRETE_ACTION + CONTROL; LIST[T] consumer-side modifier (ADR-dataflow-005)
BaseNodeSet (name, server_python, env_panel, default_per_step_budget_sec) Done plus get_tools, initialize, shutdown, get_eval_metadata
WorkspaceComponentRegistry auto-discovery Done Scans nodesets/ (role buckets: env/ method/ policy/ model/ common/ other/), policies/, nodes/; ensure_nodesets_for_graph auto-loads on graph open
Hot reload (POST /api/components/reload) Done Shuts down, re-scans, re-initializes
GenericBlockRenderer Done Renders any node from metadata; block + strip + viewer + imageGrid + note layouts
Node schema endpoint Done GET /api/components/node-schemas
Legacy base classes removed Done BaseTool, BaseSkill, BaseAgent, BasePolicy removed from bases.py

All items fully implemented.

AgentCanvas docs