Matching NodeSet

The Matching NodeSet (MatchingNodeSet, workspace/nodesets/model/model_matching.py) is the sparse counterpart to VGGT: where VGGT regresses dense geometry in one feed-forward pass, this nodeset exposes the classic sparse front-end — detect repeatable keypoints in one image, match them across two — that SLAM, relocalization, and loop-closure front-ends consume. It feeds the pySLAM nodeset's correspondence stage and any two-view geometry step. Both primitives ride Hugging Face transformers keypoint task heads, so the whole thing lives in the shared ac-fm env — no dedicated env, no external repo.

env: ac-fm (shared) · backend: transformers (AutoModelForKeypointDetection / AutoModelForKeypointMatching) · default model_id: magic-leap-community/superpoint (detect) · ETH-CVG/lightglue_superpoint (match)

1. Two primitives: detect, then match

Keypoint detection and matching are split into two nodes because they are two distinct capabilities with different arities. detect_keypoints is a one-image primitive — useful on its own (a descriptor bank for a place database, a keypoint overlay). match is the two-image primitive that answers "where does image A land in image B". A graph wires whichever it needs; matching does not require a prior detect call (the transformers matcher runs its own SuperPoint front-end internally over the pair).

2. Detect — SuperPoint

detect_keypoints wraps SuperPointForKeypointDetection (variant as model_id). The image is preprocessed by the model's own AutoImageProcessor; post_process_keypoint_detection(outputs, target_sizes=[(H, W)]) rescales the keypoints back to the original image resolution, so the returned coordinates are directly usable against the source pixels. The port carries three arrays: keypoints (N, 2), scores (N,), and descriptors (N, D) (SuperPoint's D = 256). N varies per image with the detector's response.

3. Match — LightGlue over an image pair

match wraps the AutoModelForKeypointMatching family. The transformers matcher takes an image pair, which its processor expects nested as [[image_a, image_b]]; post_process_keypoint_matching rescales to original pixels and returns only the mutually matched keypoints above threshold:

# LightGlue takes an image *pair* — the processor nests it as [[a, b]].
inputs = processor([[image_a, image_b]], return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model(**inputs)              # AutoModelForKeypointMatching
# rescale to original px, keep only mutual matches above threshold
res = processor.post_process_keypoint_matching(
    outputs, target_sizes=[[(Ha, Wa), (Hb, Wb)]], threshold=threshold)[0]
kpt0, kpt1, scores = res["keypoints0"], res["keypoints1"], res["matching_scores"]

Because the whole family shares one processor and the same post_process_keypoint_matching contract, swapping model_id alone gives the other matchers for free — SuperGlue (magic-leap-community/superglue_*) and detector-free EfficientLoFTR. keypoints0[i] and keypoints1[i] are corresponding points in image A and image B; the arrays share length M (the match count) and collapse to shape (0, 2) when nothing matches.

4. Multi-array envelope

Each port is a TEXT JSON envelope — several named arrays, each the raw C-contiguous float32 buffer base64-encoded (byte-exact across the server-mode HTTP boundary, ~4× smaller than a JSON float list), alongside model_id and the image size(s). Coordinates are absolute to the original image (image_hw = [H, W]; match also records image_hw_b for the second image):

{"model_id": "ETH-CVG/lightglue_superpoint", "image_hw": [520, 779], "image_hw_b": [520, 779],
 "keypoints0": {"shape": [60, 2], "dtype": "float32", "b64": "<C-contiguous float32 buffer>"},
 "keypoints1": {"shape": [60, 2], "dtype": "float32", "b64": "…"},
 "matching_scores": {"shape": [60], "dtype": "float32", "b64": "…"}}

Consumers read env["keypoints0"]["b64"] etc. and np.frombuffer(...).reshape(shape). The detect_keypoints port carries keypoints (N,2) + scores (N,) + descriptors (N,D).

5. Canvas Nodes

model_matching__detect_keypoints

model_matching__match

Engines are lazy singletons in a registry keyed by (kind, model_id) — detect and match load independently, so a graph using only one never pays for the other. These heads run fp32 (keypoint coordinates are precision-sensitive and the nets are tiny — no autocast); GPU inference is single-flight per engine.

6. Server Mode

parallelism = "shared" (stateless perception primitives — one server across eval workers); server_python = conda_env_python("ac-fm", "MATCHING_PYTHON"). Unlike VGGT, matching needs no dedicated env: its models are transformers keypoint task heads, so it shares the ac-fm stack with CLIP / SAM / Depth Anything. Weights (SuperPoint / LightGlue checkpoints) download lazily to the HF cache on first use — none are fetched at load. The file stays Python-3.8-parseable (an override may point at another env). Load: POST /api/components/nodesets/model_matching/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, or an image entry missing/malformed base64, yields "" on the node's output. An empty envelope is a clean "no keypoints / no matches this step" signal; consumers keep their own fallback and never receive fabricated correspondences.