Foundation Models
Foundation-model nodesets expose generic, domain-agnostic primitives (score_tokens / generate / embed). Method nodesets consume them over wires and own all task-specific glue — see TODO #56 (method vs foundation-model boundary).
The map below groups every wrapped model by capability (lane colour matches the group label). Dashed chips are common foundation models we have not wrapped yet — the current coverage gaps.
| NodeSet | File | Env | Description |
|---|---|---|---|
| BLIP-2 | workspace/nodesets/model/model_blip2.py | ac-fm | Per-view captioning (FlanT5-XL) — extracted from navgpt_mp3d_tools (2026-07-04) |
| CLIP | workspace/nodesets/model/model_clip.py | ac-fm | Language-aligned image/text embeddings — encode_image / encode_text / zero-shot classify; shared image–text space for open-vocab maps & retrieval |
| CoTracker | workspace/nodesets/model/model_cotracker.py | ac-cotracker | Point tracking (CoTracker3) — dense point tracking through a video / frame sequence with occlusion handling; track_grid / track_points |
| Depth Anything | workspace/nodesets/model/model_depth_anything.py | ac-fm | Monocular depth (Depth Anything V2) — RGB → dense per-pixel depth; relative / metric checkpoints as config |
| DepthPro | workspace/nodesets/model/model_depthpro.py | ac-fm | Zero-shot metric depth (Apple DepthPro) — RGB → dense depth in metres + recovered field of view; the absolute-scale companion to Depth Anything |
| DetAny3D | workspace/nodesets/model/model_detany3d/ | ac-detany3d | Promptable 3D detection — image + box prompts → 3D bboxes (ToolEQA perception backbone; text→box via Grounding DINO composition) |
| DINOv2 | workspace/nodesets/model/model_dinov2.py | ac-fm | Per-image pooled features (ViT-S/14-reg) — extracted from smartway_waypoint (2026-07-04) |
| Grounding DINO | workspace/nodesets/model/model_grounding_dino.py | ac-detany3d (native) · ac-fm (hf_tiny) | Open-vocab text→box detection — variant as config (native Swin-T/Swin-B or hf_tiny); serves AO-Planner + ToolEQA |
| InstructBLIP | workspace/nodesets/model/model_instructblip.py | ac-fm | Per-view scene captioning (FlanT5-XL, DiscussNav prompt) |
| Matching (SuperPoint + LightGlue) | workspace/nodesets/model/model_matching.py | ac-fm | Sparse keypoint detection + matching (SuperPoint · LightGlue/SuperGlue/LoFTR via transformers) — detect_keypoints / match; the SLAM / relocalization front-end |
| Optical Flow (RAFT) | workspace/nodesets/model/model_opticalflow.py | ac-fm | Dense optical flow (torchvision RAFT) — two frames → per-pixel motion field; raft_large / raft_small, no external weights download |
| RAM / RAM++ | workspace/nodesets/model/model_ram.py | ac-ram | Swin-L tagging; variant-keyed engines (ram/ram_plus × image_size), ordered-list + keyed-dict tools |
| SAM | workspace/nodesets/model/model_sam.py | ac-fm | Segment Anything full series (SAM 1 / 2.1 / 3) — point/box/auto/text segmentation + image embedding; variant as config, stateless server, embedding/logits dataflow ports |
| Segmentation (Mask2Former) | workspace/nodesets/model/model_segmentation.py | ac-fm | Universal segmentation (Mask2Former) — per-pixel semantic class-id maps + instance-aware panoptic maps; semantic / panoptic (SAM answers "where", this answers "what") |
| Surface Normals (Sapiens) | workspace/nodesets/model/model_normal.py | ac-fm | Surface-normal estimation (Sapiens via AutoModelForNormalEstimation) — RGB → dense per-pixel unit normals; the orientation companion to monocular depth |
| VGGT | workspace/nodesets/model/model_vggt.py | ac-vggt | Feed-forward 3D reconstruction (Visual Geometry Grounded Transformer) — N RGB views → camera poses + dense depth + world point map in one pass; reconstruct / track_points |
| SpatialBot-3B | workspace/nodesets/model/vlm_spatialbot.py | ac-ram | Depth-aware VLM caption/generate — extracted from opennav_perception (2026-07-04) |
| VLM Prismatic | workspace/nodesets/model/vlm_prismatic.py | ac-hmeqa | Generic Prismatic VLM — score_tokens, generate |
| Qwen2.5-VL | workspace/nodesets/model/vlm_qwen2_5_vl.py | ac-fm | Generic VLM generate — ReAct reasoning + VQA (ToolEQA) |