OWLv2 NodeSet

The OWLv2 NodeSet (Owlv2NodeSet, workspace/nodesets/model/model_owlv2.py) wraps OWLv2 (OWL-ViT v2) as an open-vocabulary object detector driven by a set of label queries — hand it a list of class names ("chair", "door", "table") and it returns every matching box per query. It is the zero-shot detection sibling of Grounding DINO, but from a different lineage: OWLv2 scores each candidate label independently with a CLIP-style open-vocab head, so the natural input is a label set rather than a free-text caption.

env: ac-fm · backend: transformers Owlv2ForObjectDetection + AutoProcessor · default model_id: google/owlv2-base-patch16-ensemble

1. Open-vocab by label set

OWLv2 is open-vocabulary detection like Grounding DINO, but the two ask for their targets differently, and that is the whole point of having both. Grounding DINO parses a single free-text caption and grounds spans of it to boxes. OWLv2 instead scores each candidate label independently with a CLIP-style head, so you hand it a set of queries and it returns the boxes matching each one. That makes it the natural fit for "find all of these object classes in the frame" — object goals, affordance targets, scene inventory — and because every query is scored by the same head, the resulting scores are directly comparable across the label set.

Practically: give it ["chair", "door", "table"] and you get back boxes tagged with which class each one matched, rather than having to compose and later re-parse one prose sentence. The queries input accepts a list[str], a JSON list, or a comma/newline-separated string; a blank input falls back to the configured default ("chair, door, table").

2. The detect primitive + result schema

One pure single-step primitive:

model_owlv2__detect  (image_b64: TEXT, [queries: ANY])
    → result: TEXT  (JSON {boxes:[{xyxy,score,phrase}], count, image_w, image_h, queries})

Each element of boxes is {"xyxy": [x1, y1, x2, y2], "score": <float>, "phrase": <str>}. Boxes are pixel xyxy at the input image's resolution (the post-processor is handed the decoded height/width as its target size), rounded to integers, and filtered by threshold. count is len(boxes), image_w/image_h echo the decoded frame, and queries echoes the resolved label list.

The matched phrase is looked up as queries[label_index] — the post-processor's labels tensor indexes into the flat query list. This is deliberately version-robust: it does not rely on the optional text_labels key that some transformers versions attach to the post-process output.

This result schema is identical to model_grounding_dino__detect (pixel xyxy + score + matched phrase). A graph can therefore swap detector backends — OWLv2 ↔ Grounding DINO — without touching any downstream node; the difference is only in how you phrase the target (label set vs. caption).

3. Canvas Node

model_owlv2__detect

Query resolution is a fall-through: the queries input wins, else the node's queries config, else the built-in default. The threshold config parses to a float (a bad value falls back to 0.1).

Engines are lazy singletons in a registry keyed by model_id — two OWLv2 checkpoints can coexist in one server without cross-serving detections. GPU inference is single-flight per engine (one in-flight forward bounds peak VRAM under concurrent eval workers), and the loaded model is frozen (eval(), requires_grad = False).

4. Server Mode

parallelism = "shared" (a stateless detector — one server across eval workers, which coalesce onto it); server_python = conda_env_python("ac-fm", "OWLV2_PYTHON"). Owlv2ForObjectDetection / AutoProcessor are native to the shared ac-fm FM env (transformers 5.13), so no dedicated env is needed; the checkpoints are ungated. The nodeset file stays Python-3.8-parseable (an $OWLV2_PYTHON override may point at a py3.8 env). Load: POST /api/components/nodesets/model_owlv2/load?mode=server.

5. Environment Variables

6. Degraded Mode

An empty image_b64 short-circuits before the model is touched, returning {"boxes": [], "count": 0, "error": "no image_b64"} — a clean "nothing to detect" signal, never a fabricated box. A model/processor load failure latches (_load_failed — no retry storm) and yields "" on result with a degraded self-log; consumers keep their own fallback.