SigLIP2 NodeSet

The SigLIP2 NodeSet (Siglip2NodeSet, workspace/nodesets/model/model_siglip2.py) wraps SigLIP 2 as a language-aligned visual-embedding primitive — the modern successor to CLIP. Like CLIP it lives in a shared image–text space, but it is trained with a sigmoid pairwise loss rather than CLIP's softmax contrastive loss. The practical consequence surfaces in classify: each image·label pair is scored on its own, so per-label probabilities are independent and multi-label by construction — "is this a kitchen? is this a hallway?" are answered separately, and both can be high or both low. That is exactly what open-vocabulary mapping and multi-label scene tagging want.

env: ac-fm · backend: transformers AutoModel + AutoProcessor · default model_id: google/siglip2-base-patch16-224 (768-d)

1. One shared image–text space

Image and text embeddings from the same model_id are directly comparable — both are L2-normalized, so cosine similarity is a plain dot product. A graph can wire encode_image + encode_text into its own similarity / retrieval / open-vocab-map logic without touching classify; classify is the convenience path when you just want per-label scores for one set of candidate phrases.

Labels are used verbatim — there is no hidden template. SigLIP's own eval uses "This is a photo of a {}."; supply the full prompt yourself if you want a template. Prompt engineering is the caller's decision, kept out of the primitive.

2. Encoders — projection folded into the towers

SigLIP folds the projection into each tower: the per-image / per-text embedding is the tower pooler_output — there is no separate visual_projection head as in CLIP. As with CLIP, get_image_features / get_text_features are avoided: in transformers 5.x they return the raw tower output object, not the vector. Each encoder therefore reads vision_model / text_model pooler_output directly and L2-normalizes — bit-exact with the model's own normalized image_embeds / text_embeds:

# SigLIP folds the projection into the towers: the embedding is the tower
# pooler_output, so there is no separate visual_projection head.
# get_image_features returns the raw tower output object in transformers 5.x,
# so read pooler_output directly, then L2-normalize.
feats = model.vision_model(pixel_values=pixel_values).pooler_output
feats = feats / feats.norm(p=2, dim=-1, keepdim=True)

Text is tokenised in SigLIP's training/eval regime — padding="max_length", max_length=64, truncation=True — so scores match the model's intended behaviour rather than drifting with variable-length padding.

Sigmoid, not softmax — the key difference from CLIP. classify runs the full model forward, reads logits_per_image (already scaled by the learned logit_scale and shifted by logit_bias), then applies a sigmoid — not a softmax over the label set:

out    = model(**inputs)
logits = out.logits_per_image      # (N_img, N_label), scaled + biased
probs  = torch.sigmoid(logits)     # SigLIP: per-pair sigmoid, NOT softmax

So probs[n] is a vector of independent per-pair probabilities in [0, 1] that does not sum to 1 (multi-label by construction). This contrasts with CLIP's classify, whose softmax over the label set makes the labels compete for a single unit of probability mass. top is still the single highest-scoring label per image (argmax over the row).

3. Embedding envelope

Both encoders output a TEXT JSON envelope carrying the raw C-contiguous float32 buffer base64-encoded (CLIP/DINOv2's sibling format, plus model_id + normalized self-description):

{"shape": [N, 768], "dtype": "float32", "b64": "<base64 of the C-contiguous float32 buffer>", "model_id": "google/siglip2-base-patch16-224", "normalized": true}

Byte-exact across the server-mode HTTP boundary — a plain JSON float list would round-trip every value through decimal text — and ~4× smaller. classify emits a plain JSON object instead: {"labels":[…], "probs":[N][L], "logits":[N][L], "top":[{label,prob}], "model_id":…}.

4. Canvas Nodes

model_siglip2__encode_image

model_siglip2__encode_text

model_siglip2__classify

Engines are lazy singletons in a registry keyed by model_id — two checkpoints coexist in one server without cross-serving embeddings. GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM under concurrent eval workers).

5. Server Mode

parallelism = "shared" (stateless embedding primitives — one server across eval workers); server_python = conda_env_python("ac-fm", "SIGLIP2_PYTHON"). AutoModel resolves to SiglipModel and AutoProcessor to SiglipProcessor, both native to the shared ac-fm env (Python 3.11, torch 2.8.0+cu126, transformers 5.13), so no dedicated env is needed. The nodeset file stays Python-3.8-parseable (an override may point at a py3.8 env). Load: POST /api/components/nodesets/model_siglip2/load?mode=server.

Verified 2026-07-08: AutoModel/AutoProcessor load as SiglipModel/SiglipProcessor, and an in-process forward() on CPU produced an L2-normalized (N, 768) image matrix, an (N, 768) text matrix, and classify sigmoid probabilities in [0, 1].

6. Environment Variables

7. Degraded Mode

A model/processor load failure latches (_load_failed — no retry storm); that, or an images entry missing its base64, yields "" on the respective output. Consumers keep their own fallback — an empty envelope is a clean "no embedding this step" signal, never a fabricated vector.