DetAny3D NodeSet — AgentCanvas

The DetAny3D NodeSet (ModelDetAny3DNodeSet, workspace/nodesets/model/model_detany3d/) wraps DetAny3D (Zhai 2025) — a promptable 3D detector — as a server-mode foundation-model nodeset. Like SAM, the model's native input is 2D box prompts, and since the 2026-07-05 FM-boundary reshape the nodeset exposes exactly that native capability and nothing else. It is the perception backbone of the ToolEQA port: the tooleqa method nodeset calls it via the standard server-mode HTTP route, replacing upstream's POSIX-shared-memory IPC with the platform's one IPC mechanism.

env: ac-detany3d

Primitive	Purpose
`model_detany3d__locate_3d`	Image + `[x1,y1,x2,y2]` box prompts → 3D detections via the DetAny3D head (`centers_3d` + `sizes_3d` + `rot_mat`).

What left in the reshape: locate_2d was pure GroundingDINO — that is GDINO's capability, already served by model_grounding_dino__detect (native variant, Swin-B ckpt @ 0.37/0.25 reproduces the old boxes exactly, same env and int rounding); segment was a GDINO+SAM composition with zero consumers. Their removal also drops the standalone GroundingDINO and SAM ViT-H loads (~4 GB resident) — the GDINO image features inside WrapModel are part of DetAny3D's architecture and stay. Text→box→3D is now a caller-level composition (ToolEQA's toolbox wires detect → locate_3d).

1. Architecture

Mirrors the hmeqa env-manager pattern:

DetAny3DEnvManager (singleton) holds DetAny3D's WrapModel — the whole resident surface since the 2026-07-05 reshape (the standalone GroundingDINO detector and SAM ViT-H predictor loads are gone) — pinned to a single-thread executor for GPU thread affinity.
Canvas tool node adapts the manager's predict_3d(image, boxes_2d); it carries an error output port (empty string on success) instead of raising.
Env panel — GPU selector + an explicit Init Models action for eager warm-up from the canvas.

Workspace-standalone: the DetAny3D source is vendored under model_detany3d/_vendor/ — no third_party/ imports. Model loading is identical to upstream's app_mp.py:init_models; only the IPC layer differs (HTTP server-mode instead of posix_ipc).

Moved from env/ to model/ in the 2026-07-04 nodeset-layout sweep (role test: it is a generic perception model, not an environment); node types re-prefixed model_detany3d__*. The Grounding DINO NodeSet lifts its native inference recipe from this vendored copy.

2. Canvas Nodes

Node	Inputs	Outputs
`model_detany3d__locate_3d`	`image` (ANY — RGB ndarray or path string), `boxes_2d` (ANY — list of `[x1,y1,x2,y2]` integer box prompts, or JSON string)	`centers_3d` (`[cx,cy,cz]` per prompted box), `sizes_3d` (`[sx,sy,sz]`), `rot_mat` (3×3 per box), `error`

3D outputs mirror upstream's layout: the first three columns of each predicted box are the center, the next three the size, rounded to 2 decimals. Box prompts are integers — exactly what the upstream GroundingDINO stage emitted (int-rounded), so feeding it model_grounding_dino__detect boxes reproduces the old text-path outputs bit-for-bit (equivalence-gated on captured frames, 2026-07-05).

3. Server Mode

parallelism = "shared" — DetAny3D is stateless inference, so one server serves all eval workers and the ~10 GB load stays singleton; "shared" also publishes the URL into spec._shared_urls so eval subprocesses can reach it (ADR-server-003). default_per_step_budget_sec = 30 reflects the heavy per-call cost. server_python = conda_env_python("ac-detany3d", "DETANY3D_PYTHON") — Python 3.9 + torch + flash-attn + GroundingDINO + UniDepth + SAM (built by scripts/install/install_ac_detany3d.sh).

4. Environment Variables & Weights

Variable	Default	Purpose
`DETANY3D_PYTHON`	`ac-detany3d` env python	Interpreter for the server subprocess
`DETANY3D_DATA_ROOT`	`data/detany3d`	Weights root: the DetAny3D checkpoint + `weights/sam_vit_h_4b8939.pth` (WrapModel's backbone init via `cfg.model.checkpoint`) and the vendored `demo.yaml`; `weights/groundingdino_swinb_cogcoor.pth` stays here too, now consumed by `model_grounding_dino` (ckpt config) rather than this nodeset

5. Failure Behaviour

Every tool returns structured errors through its error port (bad image input, manager-level predict errors) with empty result lists — no exceptions across the wire. The env panel's Init Models action surfaces load problems eagerly instead of on the first episode step.

6. Consumers

ToolEQA (tooleqa method nodeset) — 2D/3D localisation + segmentation tools inside the ReAct loop, alongside Qwen2.5-VL as the reasoning engine