AgentCanvas / Pages / Developer Guide / Nodesets / Model

Foundation Models

Foundation-model nodesets expose generic, domain-agnostic primitives (score_tokens / generate / embed). Method nodesets consume them over wires and own all task-specific glue — see TODO #56 (method vs foundation-model boundary).

The map below groups every wrapped model by capability (lane colour matches the group label). Dashed chips are common foundation models we have not wrapped yet — the current coverage gaps.

wrapped (29 nodesets) common FM — not yet wrapped Perception 2D Grounding DINO ac-detany3d · ac-fm SAM 1 · 2.1 · 3 ac-fm RAM / RAM++ ac-ram Segmentation ac-fm · Mask2Former SAM Video ac-fm · SAM 2 track OWLv2 ac-fm · open-vocab Geometry & 3D Depth Anything ac-fm · rel depth DepthPro ac-fm · metric Surface Normals ac-fm · Sapiens Optical Flow ac-fm · RAFT DetAny3D ac-detany3d VGGT ac-vggt SuperPoint+LG ac-fm · matching CoTracker ac-cotracker Pointmap ac-fm · Sapiens 3D Representation CLIP ac-fm · img–text DINOv2 / v3 ac-fm · hub+hf SigLIP2 ac-fm · sigmoid AIMv2 ac-fm · features Language / VLM BLIP-2 ac-fm InstructBLIP ac-fm SpatialBot-3B ac-ram Qwen2.5-VL ac-fm Prismatic ac-hmeqa Florence-2 ac-fm · unified Qwen3-VL ac-fm · img+video InternVL3 ac-fm · flagship Gemma 3 ac-fm · gated SmolVLM2 ac-fm · small/vid Common FMs not yet wrapped — the coverage gaps action / manip NoMaD / ViNT nav policy FoundationPose 6DoF pose AnyGrasp grasp detect
NodeSetFileEnvDescription
AIMv2workspace/nodesets/model/model_aimv2.pyac-fmAutoregressive ViT features (Apple AIMv2) — RGB → mean-pooled per-image embedding; a second self-supervised backbone alongside DINOv2; extract_features
BLIP-2workspace/nodesets/model/model_blip2.pyac-fmPer-view captioning (FlanT5-XL) — extracted from navgpt_mp3d_tools (2026-07-04)
CLIPworkspace/nodesets/model/model_clip.pyac-fmLanguage-aligned image/text embeddings — encode_image / encode_text / zero-shot classify; shared image–text space for open-vocab maps & retrieval
CoTrackerworkspace/nodesets/model/model_cotracker.pyac-cotrackerPoint tracking (CoTracker3) — dense point tracking through a video / frame sequence with occlusion handling; track_grid / track_points
Depth Anythingworkspace/nodesets/model/model_depth_anything.pyac-fmMonocular depth (Depth Anything V2) — RGB → dense per-pixel depth; relative / metric checkpoints as config
DepthProworkspace/nodesets/model/model_depthpro.pyac-fmZero-shot metric depth (Apple DepthPro) — RGB → dense depth in metres + recovered field of view; the absolute-scale companion to Depth Anything
DetAny3Dworkspace/nodesets/model/model_detany3d/ac-detany3dPromptable 3D detection — image + box prompts → 3D bboxes (ToolEQA perception backbone; text→box via Grounding DINO composition)
DINOv2 / v3workspace/nodesets/model/model_dinov2.pyac-fmPer-image pooled features — hub backend (byte-exact SmartWay ViT-S/14-reg) + hf backend for DINOv3 / HF DINOv2 (2026-07-08)
Florence-2workspace/nodesets/model/model_florence2.pyac-fmUnified vision (Florence-2) — one seq2seq model for caption / object detection / phrase grounding / open-vocab detection / referring segmentation / OCR via a task token; run
Grounding DINOworkspace/nodesets/model/model_grounding_dino.pyac-detany3d (native) · ac-fm (hf_tiny)Open-vocab text→box detection — variant as config (native Swin-T/Swin-B or hf_tiny); serves AO-Planner + ToolEQA
InstructBLIPworkspace/nodesets/model/model_instructblip.pyac-fmPer-view scene captioning (FlanT5-XL, DiscussNav prompt)
Matching (SuperPoint + LightGlue)workspace/nodesets/model/model_matching.pyac-fmSparse keypoint detection + matching (SuperPoint · LightGlue/SuperGlue/LoFTR via transformers) — detect_keypoints / match; the SLAM / relocalization front-end
Optical Flow (RAFT)workspace/nodesets/model/model_opticalflow.pyac-fmDense optical flow (torchvision RAFT) — two frames → per-pixel motion field; raft_large / raft_small, no external weights download
OWLv2workspace/nodesets/model/model_owlv2.pyac-fmOpen-vocabulary detection over a label set — CLIP-style per-label scoring; GroundingDINO-compatible result schema; detect
Pointmap (Sapiens)workspace/nodesets/model/model_pointmap.pyac-fmSingle-image 3D pointmap (Sapiens) — RGB → dense per-pixel (X,Y,Z) in canonical camera space + per-image scale; the monocular counterpart to VGGT's multi-view pointmap
RAM / RAM++workspace/nodesets/model/model_ram.pyac-ramSwin-L tagging; variant-keyed engines (ram/ram_plus × image_size), ordered-list + keyed-dict tools
SAMworkspace/nodesets/model/model_sam.pyac-fmSegment Anything full series (SAM 1 / 2.1 / 3) — point/box/auto/text segmentation + image embedding; variant as config, stateless server, embedding/logits dataflow ports
SAM Videoworkspace/nodesets/model/model_sam_video.pyac-fmSAM 2 video object tracking — prompt one object on frame 0 (point/box) and propagate its mask across the clip; call-local session keeps the server stateless; track
Segmentation (Mask2Former)workspace/nodesets/model/model_segmentation.pyac-fmUniversal segmentation (Mask2Former) — per-pixel semantic class-id maps + instance-aware panoptic maps; semantic / panoptic (SAM answers "where", this answers "what")
SigLIP2workspace/nodesets/model/model_siglip2.pyac-fmSigLIP 2 image–text embeddings — the CLIP sibling with a sigmoid zero-shot head (per-label, multi-label); encode_image / encode_text / classify
Surface Normals (Sapiens)workspace/nodesets/model/model_normal.pyac-fmSurface-normal estimation (Sapiens via AutoModelForNormalEstimation) — RGB → dense per-pixel unit normals; the orientation companion to monocular depth
VGGTworkspace/nodesets/model/model_vggt.pyac-vggtFeed-forward 3D reconstruction (Visual Geometry Grounded Transformer) — N RGB views → camera poses + dense depth + world point map in one pass; reconstruct / track_points
VGGT-SLAM 2workspace/nodesets/model/model_vggt_slam2/ac-vggt-slamDense RGB-only SLAM session (MIT-SPARK VGGT-SLAM 2.0, RSS 2026) — VGGT feed-forward submaps + DINOv2-SALAD loop closure + GTSAM SL(4) optimization; session verbs (reset / track / finalize / get_trajectory / get_map), the upstream evo ATE ruler (eval_trajectory), and open-set 3D object queries (query_object — PE-CLIP + SAM 3). TUM parity: fr1_xyz bit-exact vs locally-run upstream
ORB-SLAM3workspace/nodesets/model/model_orbslam3/dockerClassic feature-based RGBD visual SLAM — reset / track / trajectory session nodes; first Container Launch nodeset (server_image, no conda env)
pySLAMworkspace/nodesets/model/model_pyslam/container (agentcanvas/pyslam:cpu-fixed / :cuda)Streaming visual SLAM (Freda pySLAM, GPL-3.0) behind a rootless-Docker HTTP bridge — eleven nodes exposing pySLAM's whole surface: session (reset / track / get_trajectory / get_map / get_dense_map), stateless perception (extract_features / match_features / eval_trajectory), and neural full-surface (predict_depth / segment_semantic / reconstruct_multiview — DUSt3R/MASt3R/VGGT, GPU)
SpatialBot-3Bworkspace/nodesets/model/vlm_spatialbot.pyac-ramDepth-aware VLM caption/generate — extracted from opennav_perception (2026-07-04)
VLM Prismaticworkspace/nodesets/model/vlm_prismatic.pyac-hmeqaGeneric Prismatic VLM — score_tokens, generate
Qwen2.5-VLworkspace/nodesets/model/vlm_qwen2_5_vl.pyac-fmGeneric VLM generate — ReAct reasoning + VQA (ToolEQA)
Qwen3-VLworkspace/nodesets/model/vlm_qwen3_vl.pyac-fmGeneric VLM generate over images and video — the Qwen2.5-VL successor with a video_paths port
InternVL3workspace/nodesets/model/vlm_internvl3.pyac-fmFlagship open VLM generate over images and video — the InternVL3-*-hf transformers-native checkpoints
Gemma 3workspace/nodesets/model/vlm_gemma3.pyac-fmGoogle Gemma 3 multimodal generate over images (gated model — accept the licence once)
SmolVLM2workspace/nodesets/model/vlm_smolvlm2.pyac-fmSmall/efficient VLM generate over images and video (256M–2.2B) — the lightweight member of the palette