CoTracker NodeSet

The CoTracker NodeSet (CotrackerNodeSet, workspace/nodesets/model/model_cotracker.py) wraps CoTracker3 (Meta) as a point-tracking primitive. It is the temporal counterpart to VGGT: where VGGT's track head follows query pixels across a co-visible view set in one feed-forward pass, CoTracker follows points densely through a video / frame sequence, handling occlusion and long-range motion. It fills the gap VGGT cannot — temporal correspondence over a walk, the substrate for motion cues, visual-odometry priors, and dynamic-object handling.

env: ac-cotracker (dedicated) · backend: standalone cotracker package (git, CoTrackerPredictor) · variant: cotracker3_offline · weights: facebook/cotracker3 scaled_offline.pth (fetched lazily)

1. Two primitives: grid vs query points

track_grid is the dense "track everything" mode — it seeds a regular grid_size × grid_size grid on grid_query_frame (so N = grid_size² points) and follows them all. track_points is the targeted mode — the caller supplies exactly the pixels to follow, each as (t, x, y) = the frame the point is specified on plus its coordinates (a bare (x, y) defaults to frame 0). Both return the same envelope shape; a graph wires whichever fits — a dense grid for global motion, explicit queries to follow chosen landmarks.

2. Video in — frames, 0–255 float

The images port is an ordered list of frames (base64). They are decoded and stacked to a (1, T, 3, H, W) tensor in 0–255 range — CoTracker's own convention, with no [0,1] normalization — and fed verbatim. All frames must share one resolution (CoTracker requires it); a size mismatch or a malformed frame degrades the node to an empty output rather than guessing.

3. Weights — checkpoint, not a repo clone

CoTracker's canonical loader (torch.hub.load("facebookresearch/co-tracker", …)) clones the whole upstream repo at runtime. Since the cotracker package is already pip-installed in the env, the nodeset instead downloads only the checkpoint into the torch-hub cache and constructs the predictor directly — no repo clone, offline-friendly once cached:

# Load the offline predictor from a downloaded checkpoint — the repo is NOT cloned.
ckpt = os.path.join(torch.hub.get_dir(), "checkpoints", "scaled_offline.pth")
if not os.path.exists(ckpt):
    torch.hub.download_url_to_file(CKPT_URL, ckpt)
model = CoTrackerPredictor(checkpoint=ckpt, offline=True, window_len=60).to(device)
# video: (1, T, 3, H, W) float in 0-255 (CoTracker's own convention, no normalization)
tracks, visibility = model(video, grid_size=grid_size)   # or queries=(1,N,3) as (t,x,y)

Only cotracker3_offline (the whole-clip predictor, window_len=60) is wired; the streaming online variant is reserved. The predictor is a lazy singleton per variant, frozen and moved to the device once.

4. Multi-array envelope

Each port is a TEXT JSON envelope — the named arrays as raw C-contiguous float32 buffers, base64-encoded (byte-exact across the server-mode HTTP boundary, ~4× smaller than a JSON float list), plus the variant, the frame size image_hw = [H, W], and num_frames:

{"variant": "cotracker3_offline", "image_hw": [520, 779], "num_frames": 5,
 "tracks": {"shape": [5, 64, 2], "dtype": "float32", "b64": "<C-contiguous float32 buffer>"},
 "visibility": {"shape": [5, 64], "dtype": "float32", "b64": "…"}}

tracks is (T, N, 2) — the (x, y) of every tracked point on every frame; visibility is (T, N), a float 0/1 mask (cast from the model's bool). A point tracked off-frame keeps a predicted (extrapolated) position with visibility 0 — so out-of-bounds coordinates on invisible points are expected, not an error.

5. Canvas Nodes

model_cotracker__track_grid

model_cotracker__track_points

Engines are lazy singletons in a registry keyed by variant; GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM — CoTracker's cost scales with clip length × point count × resolution).

6. Server Mode

parallelism = "shared" (stateless tracking primitives — one server across eval workers); server_python = conda_env_python("ac-cotracker", "COTRACKER_PYTHON"). CoTracker runs in a dedicated env: it installs from git (the cotracker package is pinned to a commit, not on PyPI, and is not a transformers model), so it is kept out of the shared ac-fm stack for provenance and reproducibility. Unlike VGGT this is not a numpy<2-forced split — CoTracker is numpy-2-compatible — it is a cleanliness choice. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_cotracker/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, a frame missing/malformed base64, a frame-size mismatch, or (for track_points) absent/malformed query_points, yields "" on the node's output. An empty envelope is a clean "no tracks this step" signal; consumers keep their own fallback and never receive fabricated trajectories.