VLM Prismatic NodeSet

The VLM Prismatic NodeSet (VLMPrismaticNodeSet) wraps the Prismatic VLM (TRI-ML, 2024) as a generic foundation-model nodeset. It exposes domain-agnostic primitives that any method nodeset can wire in via canvas wires:

env: ac-hmeqa

Method nodesets stay decoupled from Prismatic — they only know about generic primitives, so swapping in another open-source VLM that exposes token-likelihoods (LLaVA, Qwen-VL, InternVL, ...) is a one-class change inside this nodeset rather than a method-side rewrite.

1. Why a foundation-model nodeset

Method nodesets (e.g. explore_eqa) used to bundle Prismatic loading, prompt formatting, and VLM-specific math inline. Three problems with that:

• Coupling: replacing the VLM forced a method-side rewrite.
• Duplication: two methods that need token-likelihood scoring would each load Prismatic separately.
• Boundary leakage: model.get_loss(...) is a Prismatic-API detail; it leaked into method code that shouldn't care.

The fix is the same pattern as policy_adapter_vlnce / policy_adapter_vla / vlm_llava (future): give the foundation model its own nodeset with a generic interface (score_tokens, generate, embed, ...). Method nodesets reach it via canvas wires, not Python imports. The boundary is enforced by the auto-host single-class-per-subprocess constraint — two nodesets can't share Python state, only wire-level data.

Codified in roadmap TODO #56 ("Clarify method vs foundation-model boundary in nodeset design").

2. Canvas Nodes

vlm_prismatic__score_tokens

Empty tokens list → empty probs. Missing image / VLM unavailable / scoring error → empty probs with a degraded/error self-log (see §5). NOTE: the checkpoint's string2idx registry covers a fixed token set (A–D, Yes/No); unregistered tokens (e.g. lowercase yes) raise inside prismatic and surface as an error.

vlm_prismatic__generate

3. Server Mode

Auto-routed to server mode by the registry because server_python points at a non-default Python: server_python = conda_env_python("ac-hmeqa", "HMEQA_PYTHON") since 2026-07-05. The ac-hmeqa env (Python 3.9 + torch 2.2.1 + Prismatic + CUDA bfloat16) is reused — no second conda env required for the VLM.

Engine registry

Engines live in a lazy registry keyed by model_id (FM template, 2026-07-05 — the former module-global _VLM_BUNDLE singleton ignored a changed id), with a load-failure latch and a single-flight GPU inference lock. The initialize() hook warms the default engine eagerly (a few minutes for prism-dinosiglip+7b on first load); subsequent score_tokens / generate calls hit the resident weights.

Parallelism

parallelism="shared" (explicit since 2026-07-05). All worker_count > 1 env subprocesses fan into the same VLM via HTTP. Future: bump to "replicated" if the platform grows multi-GPU semantics so each worker can pin its own Prismatic instance to a different GPU.

4. Environment Variables

5. Degraded Mode

When Prismatic isn't importable or the load fails (GPU unavailable, weights download failed), the engine latches and every call returns empty outputs — score_tokens → probs: [], generate → "" — with a degraded self-log entry.

The former uniform-distribution stub ([1/n, …]) was removed in the 2026-07-05 FM-template alignment: a fabricated distribution is not a model capability (mock output masquerading as signal), and it silently biased any consumer that treated the probs as real. Downstream methods now see an explicitly empty signal and decide for themselves.

6. Why Not a Commercial VLM API

The score_tokens primitive needs token-level logits for candidate strings. Commercial multimodal APIs as of 2026-05 don't expose this:

• GPT-4o, Claude Sonnet/Opus, Gemini multimodal — text-out only, no logprobs for image inputs
• OpenAI text-completion APIs expose top_logprobs but the multimodal path doesn't

Consequences:

• Methods needing calibrated confidence (explore-eqa style) require an open-source VLM — Prismatic, LLaVA, etc.
• A "commercial-VLM" path could only support generate (free-form text), not score_tokens. Those methods would have to redefine "confidence" as parsed-text heuristics ("rate 0-100..."), which is unreliable.

This is why vlm_prismatic is a server-mode local nodeset rather than an HTTP-API client.

7. Consumers

• ExploreEQA NodeSet — four score_tokens instances per step (pred A/B/C/D, rel Y/N, frontier LSV, frontier GSV)
• (future) other EQA / VLN methods that need calibrated multi-choice scoring or visual-prompted exploration

See also:

• General Policy Adapter — same singleton-loader pattern for heavy local models (VLN-CE CMA + the VLA stack)