AgentCanvas / Pages / Developer Guide / Nodesets / Model / Grounding DINO NodeSet
2026-07-06

Grounding DINO NodeSet

The Grounding DINO NodeSet (GroundingDinoNodeSet, workspace/nodesets/model/model_grounding_dino.py) wraps Grounding DINO (23.03) open-vocabulary text→box detection as a generic foundation-model nodeset. Built for the AO-Planner port's Visual Affordances Prompting layer (detect the navigable floor, then hand the boxes to SAM for a ground mask); generic enough that any open-vocab-detection method reuses it unchanged.

env: ac-detany3d (native) / ac-fm (hf_tiny)

Primitive Purpose
model_grounding_dino__detect Text→box detection: (image_b64, text_prompt?) → result JSON {boxes:[{xyxy, score, phrase}], count, image_w, image_h, text_prompt}.

1. Two variants, one schema

The variant is node config since the 2026-07-05 FM-template alignment (a select on the node UI, matching model_sam). The two variants genuinely require different interpreter envs and one server is one env, so the server env remains a deployment choice: $GROUNDING_DINO_BACKEND picks which env boots (and the variant select's default follows it). Asking a server for a variant its env cannot run raises a clear node error rather than silently ignoring the knob.

Variant Implementation Env
native (default) groundingdino-py + local .pth checkpoint — Swin-T OGC (AO-Planner's exact detector) by default; a ckpt whose filename contains swinb auto-selects the Swin-B config, and the keyed engine registry lets Swin-T and Swin-B co-host in one server. Standard transform (RandomResize 800 / ImageNet norm) → groundingdino.util.inference.predict → box_convert cxcywh→pixel xyxy (recipe shared with DetAny3D) ac-detany3d
hf_tiny HF transformers IDEA-Research/grounding-dino-tiny — the variant the retired NavGPT open_vocab_detect node ran, inference recipe moved verbatim. Its threshold= post-process kwarg requires a post-4.5x transformers signature (4.45 calls it box_threshold=, so the backend never actually worked in the previously-suggested agentcanvas env) — served from the shared ac-fm env (transformers 5.13) since 2026-07-05 ac-fm

Both backends emit the same result JSON schema, so graphs are backend-agnostic. NavGPT-style post-processing (depth-annotated object lists) lives in the pure navgpt_mp3d_tools__format_detections node, not here.

Detection defaults are AO-Planner's exact settings: caption "ground", box_threshold / text_threshold 0.4 / 0.4 (fidelity alignment 2026-06-17 — previously Swin-B @ 0.25 with "floor . ground .").

2. Canvas Nodes

model_grounding_dino__detect

Field Detail
Inputs image_b64 (TEXT), text_prompt (TEXT, optional — overrides the configured prompt)
Outputs result (TEXT — the JSON schema above)
Config variant (select native | hf_tiny — default follows $GROUNDING_DINO_BACKEND), ckpt (blank = variant default; native: .pth path, hf_tiny: HF repo id), text_prompt (default "ground"), box_threshold (0.4), text_threshold (0.4)

The former ground_mask tool is gone (evicted 2026-07-06 — cross-model composition belongs to the graph). aoplanner_ce now wires __detect → aoplanner__ground_boxes → model_sam__segment_box (sam1, ViT-H ckpt) → sample_waypoints.sam_result. Equivalence-gated on captured frames from the verified run: detect box counts 12/12, downstream candidate_pixels 12/12 exact; the mask bitmaps carry an accepted 1–3 px cross-env conv drift (SAM ViT-H moved from torch 2.1.2 to ac-fm's 2.8.0 — same class as the model_sam rewrite's documented drift, invisible to every consumer).

3. Server Mode

parallelism = "shared"; server_python follows the selected backend (2026-07-05): native → ac-detany3d, hf_tiny → ac-fm. The ac-detany3d side is a conscious reuse: that env already is the GroundingDINO host (groundingdino-py 0.4.0, torch 2.1.2+cu118, bundled Swin-T OGC config + weights) — a compiled, frozen stack that cannot move. Load: POST /api/components/nodesets/model_grounding_dino/load.

4. Environment Variables

Variable Default Purpose
GROUNDING_DINO_BACKEND native native | hf_tiny backend selection
GROUNDING_DINO_PYTHON backend-dependent: ac-detany3d (native) / ac-fm (hf_tiny) Interpreter override (applies to both backends)
GROUNDING_DINO_WEIGHTS data/detany3d/weights/groundingdino_swint_ogc.pth Native checkpoint (set a Swin-B ckpt for the stronger backbone)
GROUNDING_DINO_CONFIG GroundingDINO_SwinT_OGC.py Config basename inside the installed package
GROUNDING_DINO_HF_MODEL IDEA-Research/grounding-dino-tiny HF model id for the hf_tiny backend

5. Failure Behaviour

Missing image_b64 → empty result ({boxes: [], count: 0, error: "no image_b64"} / empty mask) without touching the model. A variant its server env cannot run raises with an actionable message (boot env / $GROUNDING_DINO_PYTHON); other load failures latch and return empty outputs with a degraded self-log (FM template, 2026-07-05). No detected boxes → empty mask_b64 with n_boxes = 0.

6. Consumers

AgentCanvas docs