NodeSet Implementations
Implementation reference for domain-specific NodeSets in workspace/nodesets/. Each page documents a single NodeSet: its purpose, canvas nodes, port schemas, and usage patterns.
For the NodeSet system design — concept, API, three deployment modes, decision guide, cookbook, file map — see NodeSets (design doc). Companion: Base Canvas Node (the class your tools subclass).
NodeSets are grouped into four families, mirrored in the sidebar: env (simulators / benchmark environments), method (agent + policy implementations), model (foundation-model wrappers — generic vision / VLM primitives), and common (reusable building blocks — tools).
Environment NodeSets
Simulators and benchmark environments — pure tools that expose observe / step / episode-control surfaces.
| NodeSet | File | Nodes | Description |
|---|---|---|---|
| Habitat | server/habitat.py |
13 | Habitat VLN-CE environment — observe, step, panorama, episode control |
| Matterport3D | server/matterport3d.py |
— | MatterSim discrete panoramic nav — env-output contract across R2R/R4R/RxR/REVERIE/CVDN/NDH |
| HM-EQA | server/hmeqa.py |
5 | HM3D semantic-mesh EQA env — free-pose teleport, episode Q/A, TSDF bounds |
| OpenEQA | server/openeqa.py |
5 | OpenEQA Episodic-Memory EQA — free-form answers scored by LLM-as-judge |
| SIMPLER | server/simpler/ |
— | SimplerEnv real-to-sim VLA eval — 25 tasks, SAPIEN + ManiSkill2_real2sim |
Method NodeSets
Agent / method implementations — reasoning-only nodesets that consume env and foundation-model tools, plus neural-policy baselines.
| NodeSet | File | Nodes | Description |
|---|---|---|---|
| NavGPT | navgpt.py |
3 | NavGPT-CE reasoning graph nodes — action format, parse, history |
| ExploreEQA | server/explore_eqa.py + server/_explore_eqa_tsdf.py |
8 | EQA exploration method — frontier visual-prompt prep, TSDF mapping, next-pose, post-hoc vote (VLM via vlm_prismatic wires) |
| General Policy Adapter | env/env_adapter/ + policy/policy_adapter_vla/ + policy_adapter_vlnce/ |
10 | Neural-policy pipeline — general env-side adapter nodeset + per-domain model-side nodesets (Pi0/SmolVLA/DP/RT-1-X · CMA/Seq2Seq) mediated by CanonicalDict |
Foundation Model NodeSets
Generic model wrappers in workspace/nodesets/model/ — domain-agnostic primitives (caption / tag / features / score_tokens / generate) that method nodesets consume over wires. Full catalog (including the pages-less wrappers) in the Foundation Models index.
| NodeSet | File | Nodes | Description |
|---|---|---|---|
| BLIP-2 | model_blip2.py |
1 | BLIP-2 FlanT5-XL per-view captioning — NavGPT's online caption chain |
| DetAny3D | model_detany3d/ |
1 | Promptable 3D detection (image + box prompts → 3D bboxes) — ToolEQA's perception backbone; text→box via Grounding DINO |
| DINOv2 | model_dinov2.py |
1 | Pooled per-image features (ViT-S/14-reg → 384-d, base64-npy envelope) — SmartWay's RGB backbone |
| Grounding DINO | model_grounding_dino.py |
1 | Open-vocab text→box detection — variant as config (native Swin-T/Swin-B | hf_tiny); AO-Planner + ToolEQA's detector |
| InstructBLIP | model_instructblip.py |
1 | Per-view scene captioning (FlanT5-XL, DiscussNav prompt) |
| RAM / RAM++ | model_ram.py |
2 | Recognize Anything Swin-L tagging — variant-keyed engines (ram / ram_plus × image_size) |
| SAM | model_sam.py |
5 | Segment Anything full series (SAM 1 / 2.1 / 3) — point/box/auto/text segmentation + image embedding, stateless ac-fm server |
| VLM Prismatic | vlm_prismatic.py |
2 | Generic Prismatic VLM foundation-model nodeset — score_tokens, generate |
| SpatialBot-3B | vlm_spatialbot.py |
2 | Depth-aware VLM — keyed-batch caption + single-view generate (Open-Nav / Three-Step Nav perception) |
| Qwen2.5-VL | vlm_qwen2_5_vl.py |
1 | Generic VLM generate (messages + image paths) — ToolEQA's ReAct engine |
Common NodeSets
Reusable building blocks — domain-agnostic primitives shared across methods.
Tools
| NodeSet | File | Nodes | Description |
|---|---|---|---|
| Basic Agent | basic_agent.py |
11 | General-purpose LLM agent tools — scratch pad, web, vision, spatial, episode |
| Geometry | common/geometry.py |
2 | Standalone utilities — distance measurement, frontier scoring |