Graph SDK Record / Replay

A graph whose env / model nodes are server-mode normally needs a GPU, one conda env per nodeset, and downloaded weights. Record / replay removes all of that from the test: g.run(record=…) tees every server-node result of one real run into a cassette file; g.run(replay=…) then runs the same graph offline, answering each server call from the recording while local pure-Python nodes run for real. The hard promise is directional: a drift in the graph's topology or port interfaces away from what was recorded fails loudly (ReplayExhausted / GraphValidationError) — but replay does not re-verify the model outputs themselves (§6). Feature-level overview: Graph SDK capability page; the proxy seam being wrapped: server-mode.

1. What it does

Every server-mode node call in a graph passes through one seam: the proxy class's forward(), generated by proxy.py · _make_forward, which POSTs to the nodeset's server subprocess and returns the decoded result. Record / replay treats that seam as the recording surface. During a real run, Recorder replaces each server-node class's forward with a tee that appends {node_type, result} to a list, in call order. On a clean finish, the list is written to a cassette together with a manifest of every recorded node type's ports and config schema. On replay, Player rebuilds those proxy classes from the saved manifests — no server, the base URL is the deliberately unresolvable http://replay.invalid — and replaces their forward with "pop the next recorded result for this node type". Everything else is the ordinary engine: the same GraphExecutor, and every local node's real forward.

The point is not to fake the graph — it is to run the real wiring, port interfaces, and local-node code against pinned server outputs. What the simulator and the model said once, they now say forever; what the graph does with it is re-executed every time.

2. The smallest replay

Three lines, runnable on a machine with none of the graph's environments (this is examples/07_habitat_sam.py's graph — Habitat in ac-vlnce, SAM in ac-fm, two GPUs' worth of servers in the real run):

g = build()                                    # the same builder the real run used
r = g.run(replay="tests/cassettes/habitat_sam_first_frame.mpk")
assert r["num_masks"] == 35                    # no GPU, no conda envs, no servers

The cassette is produced once, on a machine that does have the environments (this is a real GPU run):

# one-time, on a machine with the real envs (a real GPU run):
python examples/07_habitat_sam.py --record tests/cassettes/habitat_sam_first_frame.mpk

3. The cassette

A cassette is one file: a msgpack-encoded dict, zlib-compressed (level 9). The codec is serialization.pack_body — the same lossless encoding the proxy uses on the wire — so ndarray / torch / PIL results survive the round trip byte-exact (graph_sdk_replay.py · dump_cassette / load_cassette).

The shipped Habitat→SAM cassette (tests/cassettes/habitat_sam_first_frame.mpk) is 525 KiB compressed — four server calls, one of them a 35-mask SAM envelope.

4. Record path

Record wraps only server-node classes, only for the duration of the run, and writes only on success. Graph.run(record=…) collects the graph's server node types — every NodeDef.type containing "__" (graph_sdk.py · _drive) — and Recorder.wrap swaps each class's forward in the global NODE_HANDLERS for a tee that awaits the original and appends the result. The run itself is a completely ordinary real run: nodesets load, servers spawn, GPU admission applies. In the finally block the order is fixed: unwrap() first (restore the original forwards, so manifests are synthesised off un-mutated classes), then write() — and only if the run reached success, so a crash mid-run never leaves a truncated cassette behind.

5. Replay path

Replay never touches the workspace registry. replay=… short-circuits nodeset loading entirely (graph_sdk.py · _drive: the load branch is skipped when replay is not None), so no scan, no conda-env resolution, no subprocess. Player.install then does two things (graph_sdk_replay.py · Player.install):

1. Rebuild missing proxy classes — for each saved manifest, proxy.generate_proxy_nodes("http://replay.invalid", manifest) produces the same proxy classes a live server would have yielded, and any class the cassette needs that isn't already in NODE_HANDLERS is register_node()'d. The bogus URL is intentional: the rebuilt forward is replaced in step 2, so nothing ever dials it.
2. Override forward on every replayed node type — rebuilt or pre-existing — with a closure that pops the next result from that node type's FIFO deque, or raises ReplayExhausted when the queue is empty.

Player.uninstall runs in the finally: pre-existing classes get their original forward back; classes the Player registered are removed from NODE_HANDLERS.

6. What fails loudly — and what doesn't

Replay is an order-based contract: it checks that the graph fires the same server-call sequence it fired when recorded, not that the inputs are the same. Concretely:

7. Where it deviates from the mental model

"A recording of the run" suggests more fidelity than the mechanism provides. Four seams:

8. CI usage

The recorded cassette is committed (tests/cassettes/habitat_sam_first_frame.mpk), and the guard test skips rather than fails when it is absent — so the suite passes with or without the artifact, and runs the full offline replay when it is there (it does, in CI):

@pytest.mark.skipif(not _cassette_path().exists(), reason="habitat-sam cassette not present")
def test_habitat_sam_replay():
    g = _load_example07().build()
    r = g.run(replay=str(_cassette_path()))
    assert r["num_masks"] == 35        # recorded SAM output flows through the real overlay

Regeneration is deliberate, not automatic: rerun the example's --record flag on a machine with the real environments whenever the recorded interfaces genuinely change, and commit the new cassette. A version bump of CASSETTE_VERSION invalidates every existing cassette at load time.

9. Key files