RAM NodeSet
The RAM NodeSet (RAMPerceptionNodeSet, workspace/nodesets/model/model_ram.py) wraps Recognize Anything (23.06) Swin-L image tagging — both the original RAM and the RAM++ variant — as a generic foundation-model nodeset. Originally built for DiscussNav's per-direction tagging; extended in the TODO #56 boundary pass (2026-07-04) with a variant-keyed engine that replaced the RAM copies embedded in opennav_perception and smartway_perception.
env: ac-ram
| Primitive | Purpose |
|---|---|
model_ram__tag_panorama |
Ordered tagging: list of {dir_id, rgb_base64} → tags_per_dir (LIST[TEXT], aligned 1:1; separators stripped — DiscussNav convention). |
model_ram__tag_views |
Keyed-batch tagging: {key: {rgb_base64}} → tags ({key: tag string}); per-node config picks variant / resolution / separator mode. |
1. Variant-keyed engine
Engines are lazy singletons in a registry keyed (variant, ckpt, image_size). variant picks the factory (ram.models.ram vs ram.models.ram_plus — same package, same transform/inference API), so Open-Nav's ram@224 and SmartWay's ram_plus@384 co-reside in one server subprocess (~2 models' VRAM).
| Variant | Default checkpoint | Used by |
|---|---|---|
ram |
data/opennav/ram_swin_large_14m.pth ($RAM_CKPT) |
DiscussNav (@384), Open-Nav / Three-Step Nav (@224) |
ram_plus |
data/smartway/ram_plus/ram_plus_swin_large_14m.pth ($RAM_PLUS_CKPT) |
SmartWay (@384) |
Checkpoint paths resolve against a repo-root walk-up to the data/ anchor — a fixed ../../.. breaks when the file is served from a workspace overlay copy at a different depth. initialize(ram_ckpt=…, ram_plus_ckpt=…) can override the per-variant defaults as nodeset kwargs.
Stateless — the former content-hash tag cache was removed in the 2026-07-05 FM-template alignment: engines hold loaded weights only, and reuse of byte-identical recurring views is a graph-level decision (prototype ruling). Load failure latches (empty outputs + degraded self-log) and GPU inference is single-flight per engine.
2. Canvas Nodes
model_ram__tag_panorama
| Field | Detail |
|---|---|
| Inputs | views (ANY — ordered list of {dir_id, rgb_base64}, e.g. 12 directions) |
| Outputs | tags_per_dir (LIST[TEXT] — aligned 1:1 with views) |
| Config | variant (ram | ram_plus), image_size (384 | 224), keep_separators (toggle), ckpt — same surface as tag_views since 2026-07-05 (previously hardcoded ram@384); defaults preserve the DiscussNav behaviour |
Source: DiscussNav's per-heading ram_img_tagging inside observe_environment. Each entry is the space-joined tag string after stripping RAM's " |" separators.
model_ram__tag_views
| Field | Detail |
|---|---|
| Inputs | views (ANY — {key: {rgb_base64}} keyed dict; keys opaque, insertion order preserved) |
| Outputs | tags (ANY — {key: tag string}) |
| Config | variant (ram | ram_plus), image_size (224 | 384, default 384), keep_separators (toggle, default off), ckpt (optional override) |
keep_separators is an LLM-visible fidelity detail: Open-Nav passes the RAM output raw — "tag | tag" separators included — so its graphs set the toggle on; the default (off) strips to the DiscussNav convention.
3. Server Mode
server_python defaults to the ac-ram env (Python 3.10, torch 2.4.1, transformers <4.40, recognize-anything from upstream git — ships both the ram and ram_plus factories). The dedicated env exists because the RAM library's apply_chunking_to_forward import breaks under newer transformers releases. Override with $RAM_PERCEPTION_PYTHON (resolved via conda_env_python since 2026-07-05; parallelism="shared" is now explicit). Load: POST /api/components/nodesets/model_ram/load?mode=server.
4. Environment Variables
| Variable | Default | Purpose |
|---|---|---|
RAM_PERCEPTION_PYTHON |
ac-ram env python |
Interpreter for the server subprocess |
RAM_CKPT |
data/opennav/ram_swin_large_14m.pth |
ram variant checkpoint |
RAM_PLUS_CKPT |
data/smartway/ram_plus/ram_plus_swin_large_14m.pth |
ram_plus variant checkpoint |
5. Failure Behaviour
Per-view tag exceptions are caught and logged; that view's slot/key gets "" and the rest proceed (alignment / key coverage preserved). A model-load failure latches and every call returns empty outputs with a degraded self-log entry. An unknown variant raises at engine lookup. Empty views → empty output without touching the model.
6. Consumers
- DiscussNav —
tag_panoramaper 12-direction sweep, paired 1:1 with InstructBLIP captions - Open-Nav / Three-Step Nav —
tag_views(ram@ 224,keep_separatorson) over the candidate views; replacedopennav_perception__tag - SmartWay —
tag_views(ram_plus@ 384) over the predictor's candidates dict; replacedsmartway_perception__tag— the one cross-env move of the extraction (ac-smartway → ac-ram), output parity verified 12/12