Optical Flow NodeSet

The Optical Flow NodeSet (OpticalFlowNodeSet, workspace/nodesets/model/model_opticalflow.py) wraps RAFT (Recurrent All-Pairs Field Transforms, shipped in torchvision.models.optical_flow) as a dense motion primitive: a pair of consecutive RGB frames in, a per-pixel displacement field out. It fills the optical-flow coverage gap — the frame-to-frame motion cue that ego-motion estimation, moving-object detection, and short-horizon dynamics all build on, without a hand-tuned pyramid.

env: ac-fm (shared FM env) · backend: torchvision RAFT (no HF download — weights ship with torchvision) · variants: raft_large (default) / raft_small

1. One primitive: flow A → B

The tool takes exactly two frames — image_a and image_b, the "before" and "after" — and returns the motion carrying each pixel of A to its match in B. This is the atomic optical-flow operation; a graph that wants flow over a clip loops it across consecutive pairs. Keeping the node to a single pair (rather than a whole video) mirrors the house "one pure primitive" style and keeps peak memory bounded to one frame pair.

2. Frames in — same resolution, uint8

Each port accepts a {rgb_base64} dict (or a raw base64 string). Both frames are decoded to HWC uint8 RGB arrays and must share one resolution — a size mismatch or a malformed frame degrades the node to an empty output rather than guessing. The checkpoint's own preprocessing transform (weights.transforms()) converts to float and normalizes to [-1, 1]; it does not resize.

3. Weights & the divisible-by-8 rule

RAFT requires spatial dims divisible by 8, so each frame is zero-padded up to the next multiple of 8, run, then the flow is cropped back to the original (H, W) — preserving true pixel scale (no resize-and-rescale). The model returns its iterative-refinement list; the last entry is the final flow:

# weights ship with torchvision — no HF download, offline-friendly.
weights = Raft_Large_Weights.DEFAULT          # or Raft_Small_Weights
net = raft_large(weights=weights).eval().to(device)
transform = weights.transforms()          # normalize to [-1,1], no resize
# … pad both frames to a multiple of 8 …
flow = net(frame_a, frame_b)[-1]       # final iterate, (1, 2, Hpad, Wpad)
flow = flow[:, :, :H, :W]              # crop padding back

raft_large is the accurate default; raft_small is ~10× smaller and faster (coarser). Each variant is a lazy singleton in a registry, frozen and moved to the device once; inference is single-flight per engine.

4. Flow envelope

The output is a TEXT JSON envelope — the flow as a raw C-contiguous float32 buffer, base64-encoded (byte-exact across the server-mode HTTP boundary, ~4× smaller than a JSON float list):

{"shape": [480, 640, 2], "dtype": "float32", "b64": "<C-contiguous float32 buffer>", "variant": "raft_large"}

flow[y, x] is the displacement in pixels carrying pixel (x, y) of frame A to its match in frame B: channel 0 = dx (+ right), channel 1 = dy (+ down). The field is at the original input resolution.

5. Canvas Node

model_opticalflow__estimate_flow

6. Server Mode

parallelism = "shared" (a stateless flow estimator — one server across eval workers); server_python = conda_env_python("ac-fm", "OPTICAL_FLOW_PYTHON"). Optical flow runs in the shared ac-fm env — torchvision RAFT is native there and needs no external install or extra weights download. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_opticalflow/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, a missing/malformed frame, or a resolution mismatch between A and B yields "" on the node's output. An empty envelope is a clean "no flow this step" signal; consumers keep their own fallback and never receive a fabricated motion field.