DINOv2 NodeSet — AgentCanvas

The DINOv2 NodeSet (DinoV2NodeSet, workspace/nodesets/model/model_dinov2.py) wraps DINOv2 (23.04) ViT-S/14 with registers as a generic per-image feature primitive. It was extracted from the smartway_waypoint engine — where DINOv2 lived as the RGB backbone — in the TODO #56 boundary pass (2026-07-04, explicit user decision: the task-specific DDPPO depth encoder stays method-internal, the DINOv2 backbone does not).

env: ac-fm

Primitive	Purpose
`model_dinov2__extract_features`	Pooled embedding per view: ordered list of `{rgb_base64}` → `features` (TEXT — base64-npy JSON envelope of the (N, 384) float32 matrix).

1. Extraction & fidelity

Load + forward are byte-faithful to the SmartWay engine (upstream Policy_ViewSelection_VLNBERT.py:111 + base_il_trainer.py:356) — frozen/eval hub model, HF image processor, one batched no-grad forward:

model = torch.hub.load("facebookresearch/dinov2", "dinov2_vits14_reg").to(device)
model.eval()
for p in model.parameters():
    p.requires_grad = False
processor = AutoImageProcessor.from_pretrained("facebook/dinov2-small")

Ordering semantics — DINOv2 is a per-image ViT with no cross-sample ops, so features computed here in input order equal the ones the old engine computed after its clockwise reorder. The consumer owns any reordering (SmartWay's CW remap now happens inside the waypoint predictor, on the received features). Verified bitwise-equal through the envelope + remap against the old in-engine forward.

2. Feature envelope

The output is a TEXT JSON envelope carrying the raw C-contiguous float32 buffer base64-encoded:

{"shape": [12, 384], "dtype": "float32", "b64": "<base64 of the C-contiguous float32 buffer>"}

Byte-exact across the server-mode HTTP boundary — a plain JSON float list would round-trip every value through decimal text — and ~4× smaller.

3. Canvas Nodes

`model_dinov2__extract_features`

Field	Detail
Inputs	`views` (ANY — ordered list of `{rgb_base64}` dicts, e.g. 12 directions)
Outputs	`features` (TEXT — the §2 envelope; `""` on degraded)
Config	`hub_model` (default `dinov2_vits14_reg`), `processor_id` (default `facebook/dinov2-small`)
Backend call	one batched `model(pixel_values)` under `no_grad` → `(N, D)` float32, input order preserved

Engines are lazy singletons in a registry keyed (hub_repo, hub_model, processor_id) — two configs coexist in one server without cross-serving features. GPU inference is single-flight per engine since 2026-07-05 (one in-flight forward bounds peak VRAM under concurrent eval workers).

4. Server Mode

parallelism = "shared" (stateless extractor — one server across eval workers); server_python = conda_env_python("ac-fm", "DINOV2_PYTHON") since 2026-07-05.

Numeric parity with the old ac-smartway hosting is byte-exact, verified in a two-part probe: the ViT forward is bit-identical across torch 2.1.1→2.8.0, and the processor is pinned to the PIL backend with use_fast=False — transformers 5.x flipped AutoImageProcessor's default to a torchvision implementation whose resize numerics differ (the kwarg is a no-op on 4.x, so an override env stays byte-aligned too). The nodeset file must stay Python-3.8-parseable (an override may point at a py3.8 env). Load: POST /api/components/nodesets/model_dinov2/load?mode=server.

5. Environment Variables

Variable	Default	Purpose
`DINOV2_PYTHON`	`ac-fm` env python	Interpreter for the server subprocess

6. Degraded Mode

A hub/processor load failure latches (_load_failed — no retry storm); that, or any view missing rgb_base64, yields features = "". Consumers keep their own fallback — the SmartWay waypoint predictor degrades to a zeros (12, 384) RGB embedding with a logged warning, which is exactly the old in-engine failure behaviour.

7. Consumers

SmartWay — smartway_ce.json / smartway_mono_ce.json: panorama_rgbd.views → model_dinov2__extract_features → smartway_waypoint__predict.rgb_features. The consumer port is required, not optional — an optional port would not gate firing and silently races to the zeros fallback.