CLIP NodeSet

The CLIP NodeSet (ClipNodeSet, workspace/nodesets/model/model_clip.py) wraps CLIP as a language-aligned visual-embedding primitive. Where DINOv2 gives a self-supervised per-image feature with no notion of language, CLIP gives an embedding that lives in a shared image–text space — a picture and the phrase describing it land near each other. That shared space is the geometry behind open-vocabulary maps (VLMaps / ConceptFusion), zero-shot retrieval, and CLIP-reward models.

env: ac-fm · backend: transformers CLIPModel + CLIPProcessor · default model_id: openai/clip-vit-base-patch32 (512-d)

1. One shared image–text space

Image and text embeddings from the same model_id are directly comparable — both are L2-normalized, so cosine similarity is a plain dot product. A graph can wire encode_image + encode_text into its own similarity / retrieval / open-vocab-map logic without touching classify; classify is the convenience path when you just want per-label probabilities for one set of candidate phrases.

Labels are used verbatim — there is no hidden "a photo of a {}" template. Prompt engineering is the caller's decision, kept out of the primitive.

2. Encoders — tower + projection

Each encoder runs the relevant CLIP tower, applies the projection head, then L2-normalizes. The projection is done explicitly rather than via get_image_features / get_text_features: in transformers 5.x those return the raw vision/text output (pre-projection), not the projected embedding. This path is bit-exact with the model's own normalized image_embeds / text_embeds:

# transformers 5.x: get_image_features returns the raw vision output (pre-projection),
# so project explicitly through the head, then L2-normalize.
# Bit-exact with the model's own normalized image_embeds (verified 2026-07-07).
vision_out = model.vision_model(pixel_values=pixel_values)
feats = model.visual_projection(vision_out.pooler_output)
feats = feats / feats.norm(p=2, dim=-1, keepdim=True)

classify instead runs the full model forward and reads logits_per_image (already scaled by the learned logit_scale), then a softmax over labels — so probs is genuine CLIP zero-shot behaviour, not a bare cosine.

3. Embedding envelope

Both encoders output a TEXT JSON envelope carrying the raw C-contiguous float32 buffer base64-encoded (DINOv2's sibling format, plus model_id + normalized self-description):

{"shape": [N, 512], "dtype": "float32", "b64": "<base64 of the C-contiguous float32 buffer>", "model_id": "openai/clip-vit-base-patch32", "normalized": true}

Byte-exact across the server-mode HTTP boundary — a plain JSON float list would round-trip every value through decimal text — and ~4× smaller. classify emits a plain JSON object instead: {"labels":[…], "probs":[N][L], "logits":[N][L], "top":[{label,prob}], "model_id":…}.

4. Canvas Nodes

model_clip__encode_image

model_clip__encode_text

model_clip__classify

Engines are lazy singletons in a registry keyed by model_id — two checkpoints coexist in one server without cross-serving embeddings. GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM under concurrent eval workers).

5. Server Mode

parallelism = "shared" (stateless embedding primitives — one server across eval workers); server_python = conda_env_python("ac-fm", "CLIP_PYTHON"). CLIPModel / CLIPProcessor are native to the shared ac-fm env (transformers 5.13), so no dedicated env is needed. The nodeset file stays Python-3.8-parseable (an override may point at a py3.8 env). Load: POST /api/components/nodesets/model_clip/load?mode=server.

6. Environment Variables

7. Degraded Mode

A model/processor load failure latches (_load_failed — no retry storm); that, or an images entry missing its base64, yields "" on the respective output. Consumers keep their own fallback — an empty envelope is a clean "no embedding this step" signal, never a fabricated vector.