AIMv2 NodeSet

The AIMv2 NodeSet (Aimv2NodeSet, workspace/nodesets/model/model_aimv2.py) wraps AIMv2 (Apple) as a second self-supervised per-image feature backbone alongside DINOv2. It fills the same graph role — a generic per-image descriptor with no language head — but comes from a different pre-training recipe, so its feature geometry differs and it is worth having both when a downstream head is sensitive to the backbone.

env: ac-fm · backend: transformers AutoModel (Aimv2VisionModel) + AutoImageProcessor · default model_id: apple/aimv2-large-patch14-224 (1024-d)

1. A second self-supervised backbone

AIMv2 is trained autoregressively: a multimodal decoder predicts image patches (and text), a different objective from DINOv2's self-distillation. The two backbones therefore land in different feature spaces even though both are label-free per-image ViTs with no notion of language. Neither is universally better — which one transfers best depends on the downstream head — so the graph keeps both available and picks per experiment.

Like DINOv2, AIMv2 has no language head: it produces a visual descriptor, not a shared image–text embedding. For language-aligned embeddings (open-vocab maps, zero-shot retrieval) reach for CLIP instead.

2. Mean-pooled descriptor & envelope

The AIMv2 vision tower emits per-patch tokens (N, P, D) with no CLS / pooler token. The per-image descriptor here is therefore the mean over the patch tokens of last_hidden_state — the standard global-feature reduction for a CLS-less ViT:

feats = model(pixel_values=pixel_values).last_hidden_state  # (N, P, D)
feats = feats.mean(dim=1)  # (N, D)

Features are not L2-normalized — they are raw pooled activations. Normalize downstream if a cosine metric is wanted. The output is a TEXT JSON envelope carrying the raw C-contiguous float32 buffer base64-encoded (DINOv2/CLIP's sibling format, plus model_id + a pooled self-description):

{"shape": [N, 1024], "dtype": "float32", "b64": "<base64 of the C-contiguous float32 buffer>", "model_id": "apple/aimv2-large-patch14-224", "pooled": "mean"}

Byte-exact across the server-mode HTTP boundary — a plain JSON float list would round-trip every value through decimal text — and ~4× smaller. The large default gives D = 1024.

3. Canvas Nodes

model_aimv2__extract_features

Engines are lazy singletons in a registry keyed by model_id — two checkpoints coexist in one server without cross-serving features. GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM under concurrent eval workers).

4. Server Mode

parallelism = "shared" (stateless extractor — one server across eval workers); server_python = conda_env_python("ac-fm", "AIMV2_PYTHON"). AutoModel loads the AIMv2 checkpoint as an Aimv2VisionModel, and both it and AutoImageProcessor are native to the shared ac-fm env (transformers 5.13), so no dedicated env is needed and the checkpoints are ungated. The nodeset file stays Python-3.8-parseable (an override may point at a py3.8 env). Load: POST /api/components/nodesets/model_aimv2/load?mode=server.

5. Environment Variables

6. Degraded Mode

A model/processor load failure latches (_load_failed — no retry storm); that, or an images entry missing its base64, yields features = "". An empty envelope is a clean "no features this step" signal, never a fabricated vector — consumers keep their own fallback.