Wire Type System

Every port on every node carries a wire type — a short string like IMAGE, TEXT, or CONTROL. The type does four jobs: it pins one canonical format (dtype / shape / range / units) so a consumer never defensively normalises, it decides which ports may connect, it picks how the value is serialised when it crosses a process boundary, and it colours the handle on the canvas. This page is the authority on the type system; for the EdgeDef/PortDef schema that holds these strings, see Graph System §2.1.

1. What a wire type is

A wire type is a label on a port, nothing more — there is no class hierarchy, no generics, no inheritance. The vocabulary is a flat set of nine current inner types (plus three deprecated names kept for backward-compat) and one modifier (LIST[T]). A node author writes the label as a plain string in a PortDef; the framework reads it at four moments — edge creation (frontend compatibility), graph load (backend compatibility, §7), transport (serialisation), and render (colour). Values on the wire are not runtime-checked against their type by default: a node can technically emit anything (§7), so the canonical format (§3) is a producer-side contract, not a guard.

2. The compatibility rule, in one sentence

The full algebra and the source function are in §6; the surprising part — where this rule is actually enforced — is in §7.

3. The registry at a glance

The canonical set lives in agentcanvas/backend/app/standard/wire_types.py as WIRE_TYPES; the format column is the machine-readable WIRE_FORMAT_SPEC in the same file. Nine current inner types, plus the LIST[T] modifier:

Deprecated (still in WIRE_TYPES so legacy graphs load; see §13): ACTION (→ DISCRETE_ACTION via WIRE_TYPE_ALIASES), and the dead composites OBSERVATION / STEP_RESULT (no nodeset emits them — flat rgb/depth ports + the gym tuple reward/terminated/truncated/info won).

That is the whole type system. The rest of this page is detail: what each type's value looks like (§4), how the list modifier behaves (§5), the exact compatibility algebra and where it's enforced (§6–§7), how values cross process boundaries (§8), how handles are coloured (§9), how this relates to the separate state value types (§10), and the deprecation/migration plan (§13).

4. Per-type semantics

Most types are obvious from the table. The two action types and the two fixed-shape dicts are worth pinning down:

POSE     = {"position": [x, y, z], "orientation": [qx, qy, qz, qw]}
CONTROL  = {"pos": [dx, dy, dz], "rot": [rx, ry, rz], "gripper": 1.0}  # rot = axis-angle rad; gripper +1 open / -1 close
METRICS  = {"spl": 0.42, "sr": 1.0, "ndtw": 0.65, "sdtw": 0.65}

DISCRETE_ACTION is an int or str index/id into the env's declared valid set — the env owns the set, the type no longer hardwires 0–3. So discrete Habitat nav (0=STOP, 1=FORWARD, 2=TURN_LEFT, 3=TURN_RIGHT), RxR (0–5), and MP3D panoramic viewpoint-id selection all share this one type. CONTROL is a named-field continuous motor command (delta end-effector + gripper, optionally absolute joint positions) — deliberately a self-describing dict rather than a bare 7-vector, so it isn't a raw-ndarray escape. POSE doubles as a "go-to-pose / teleport" action (greedy-follower nav, HM-EQA free-pose), and TEXT doubles as an EQA free-text answer. The STATE → POSE rename (ADR-dataflow-004) freed the word “state” to mean a stored memory slot (state containers). The deprecated composites OBSERVATION (IMAGE+DEPTH) and STEP_RESULT (OBSERVATION+POSE+BOOL+METRICS?) are documented in §13.

5. The LIST[T] modifier

LIST[T] (ADR-dataflow-005) is a modifier, not a new type: any inner T can be wrapped so a port carries list[T]. It exists for multi-image VLM calls (MapGPT), multi-LLM debate transcripts (DiscussNav), and population fan-in for search operators.

The design rule is consumer-side coercion: producers stay scalar (a node that emits one T keeps its port T), and a consumer opts in by declaring its input LIST[T]. Whatever arrives, the executor guarantees the handler sees a list[T]:

The coercion happens in exactly one place — the executor's port-binding seam, _route_value_to_port in graph_executor.py. Producers, viewers, state containers, and serializers are all unchanged. Fan-in order follows the graph JSON's edge order, which keeps positional prompt references (“Image 3 is the third viewpoint”) stable across runs.

Use it when a node genuinely produces or consumes many items per firing. Don't wrap a scalar producer “just in case” — the executor wraps at the consumer, so it's redundant. Nested LIST[LIST[T]] is not supported; flatten in the producer or route through a composite.

Declaring a list port in Python is just the wrapped string; on the canvas, the PortListEditor offers an inner type plus a list checkbox, and the saved wire_type becomes LIST[<inner>]. List handles render with a doubled outer ring in the inner type's colour (§9).

6. The compatibility algebra in full

validate_edge_wire_type(source_type, target_type) → (ok, reason) in graph_def.py is the canonical rule. Six cases, in order:

Note the asymmetry between cases 4 and 5: widening one value into a singleton list is safe and automatic, but narrowing a list to a scalar would silently drop data, so it is refused.

7. Validation — and where it actually happens

What enforces what:

• Canvas drag (frontend, live): isCompatibleWireConnection (layoutUtils.ts) runs inside useFlowStore.onConnect before an edge enters the store. On a mismatch the drag is silently dropped (React Flow snaps the wire back) — it mirrors the §6 algebra exactly, stripping LIST[…] before comparing inner types. One asymmetry vs the backend: it does not canonicalise deprecated aliases, so a legacy ACTION → migrated DISCRETE_ACTION edge — which the backend accepts via canonical_wire_type (§13) — is rejected on the canvas during the migration window.
• Graph load (backend, hard 400): validate_graph_connectivity (graph_def.py) runs at POST /api/navigate/run and POST /api/eval/v2/start. Alongside its required-but-unwired-port check it now calls validate_edge_wire_types, raising HTTP 400 on a mismatch. At load time only builtin/iterIn types are resolvable (nodesets are loaded just after), so this layer hard-checks the builtin/iterIn edges.
• Runtime (backend, warn-only — staged): after ensure_nodesets_for_graph populates NODE_HANDLERS, the run/eval paths re-run validate_edge_wire_types over the whole graph (nodeset env↔method edges now resolvable) and log any mismatch. This is warn-only pending a clean-sweep before promotion to a hard 400.
• Standalone dev tool: python -m app.tools.validate_graph <name|--all> statically imports each nodeset module to read its declared ports (no simulator, no GPU, no backend — heavy runtimes load lazily) and runs the same wire_type_report core. The dev endpoint POST /api/graphs/validate exposes the same check (non-raising; optional ?ensure loads nodesets for full coverage).
• Runtime value guards (opt-in): the is_valid_* guards in wire_types.py are advisory — a handler may call them to assert its inputs, but nothing enforces them automatically.

is_valid_image(obj)            # ndarray, ndim==3, shape[2]==3
is_valid_depth(obj)            # ndarray, ndim==2
is_valid_discrete_action(obj)  # int | str — env owns the valid set
is_valid_control(obj)          # dict with pos/rot/gripper/joint_position
is_valid_pose(obj)             # dict with "position" and "orientation"
is_valid_list_of("IMAGE", obj) # list whose items each pass is_valid_image

8. Serialisation across boundaries

The same wire type takes a different concrete form on each of three transports. Authors never serialise by hand — the framework converts at the boundary, keyed on wire_type:

In-process there is no serialisation at all — the executor moves result["rgb"] into the next node's pending inputs by reference. Over WebSocket and HTTP, JSON can't hold a numpy array, so two helper layers handle it:

• Display codecs (lossy, display-only) — app/standard/wire_types.py: image_to_base64 / base64_to_image, depth_to_base64 / base64_to_depth, and serialize_for_display(wire_type, value) which viewer sinks use to build viewer_data payloads. depth_to_base64 normalises float32 metres to a 0–255 heatmap — fine for the viewer, never for the data path.
• HTTP transport (data path, lossless) — app/server/serialization.py: serialize_value / deserialize_value round-trip wire values for server-mode /call/{fn} traffic. IMAGE goes to a base64 PNG (lossless for uint8, compresses natural frames); everything else numpy — including DEPTH — rides the {"__ndarray__": …, "dtype": …, "shape": …} marker via _make_json_safe / _restore_ndarrays (np.frombuffer on the receiver), 4× smaller and lossless versus nested JSON lists.

The single rule behind all of this: one canonical format per type, and the data path is always lossless. Native Python in-process, JSON-safe over the network (PNG for images, __ndarray__ for every other array), lossy encodings confined to the display path — so the same handler code runs, with the same numeric values, whether a node is in-process or remote.

9. Canvas colours

A handle's colour is its wire type, not its node category — don't confuse the two: handle = wire type, node frame = category (the frame tint comes from CATEGORY_STYLES in the same file). The canonical map is WIRE_COLORS in layoutUtils.ts; getWireColor strips a LIST[…] wrapper before lookup, so a LIST[IMAGE] handle uses the IMAGE colour with a doubled ring (§5):

Any type without an entry falls back to the ANY slate. (POSE and STEP_RESULT were added to WIRE_COLORS on 2026-06-13 — before that, the STATE → POSE rename had left POSE handles rendering as slate.)

10. Wire types vs state value types

A wire type travels through an edge once per firing — ephemeral, per-step. A state value type is the type of a slot in a state container — persistent memory that accumulates across iterations. The two registries overlap but are not the same: STATE_VALUE_TYPES drops the raw frame types (a container holds memory, not pixels) and adds domain memory types (POINTCLOUD, OCCUPANCY_MAP, EMBEDDING, EPISODE_CONTEXT).

STATE_VALUE_TYPES = {                  # state_containers.py — memory, not raw frames
    "TEXT", "BOOL", "METRICS", "POSE",
    "POINTCLOUD", "OCCUPANCY_MAP", "EMBEDDING", "EPISODE_CONTEXT",
    "ANY",
}
_REMOVED_VALUE_TYPES = {               # rejected at build_container() with a ValueError
    "STATE", "IMAGE", "DEPTH", "ACTION", "OBSERVATION", "STEP_RESULT",
}

Depth on the reducers, lifetimes, and grants is in State Containers; this contrast is all that belongs here.

11. Adding a new wire type

Three touch-points, all mechanical:

1. Register the constant in WIRE_TYPES and add its canonical-format line to WIRE_FORMAT_SPEC (app/standard/wire_types.py) — the spec is the contract a node author trusts. For LIST[T] support you do nothing — the modifier wraps any inner type automatically.
2. Colour it: add a key to WIRE_COLORS (layoutUtils.ts), else handles render slate. Add it to the PortListEditor dropdown if authors should be able to pick it.
3. Serialise it (only if it isn't already JSON-safe): add a branch to serialize_for_display / the HTTP transport, and optionally a is_valid_<type> guard.

12. File reference

13. Deprecation & migration plan

The v2 catalog change (2026-06-13) is additive and backward-compatible — the new types ship and the old names stay registered so every existing graph still loads. Nodeset/graph migration runs in later batches.

Staged removal, per batch:

1. Backend catalog + spec (done 2026-06-13): add DISCRETE_ACTION / CONTROL + WIRE_FORMAT_SPEC, alias ACTION, loosen/extend the is_valid_* guards, canonicalise in validate_edge_wire_type.
2. Nodeset PortDefs: flip ACTION → DISCRETE_ACTION across env/policy/method nodesets; introduce CONTROL on the VLA envs (env_simpler, env_libero) and the policy-adapter pipeline (env_adapter + policy_adapter_vla) in place of the JSON-in-TEXT action smuggling; converge each producer onto the §3 canonical formats.
3. Frontend: add DISCRETE_ACTION / CONTROL to WIRE_COLORS + the PortListEditor dropdown (§9).
4. Drop the deprecated names from WIRE_TYPES + WIRE_TYPE_ALIASES once no PortDef references them; promote the runtime whole-graph check (§7) from warn to hard 400 after a clean sweep.