BLIP-2 NodeSet — AgentCanvas

The BLIP-2 NodeSet (Blip2NodeSet, workspace/nodesets/model/model_blip2.py) wraps BLIP-2 (23.01) FlanT5-XL captioning as a generic foundation-model nodeset. It was extracted from navgpt_mp3d_tools — where the caption node lived as a method node — in the TODO #56 boundary pass (2026-07-04): per-view captioning is a generic vision primitive; NavGPT's task glue stayed behind.

env: ac-fm

Primitive	Purpose
`model_blip2__caption`	Per-view captioning: ordered list of `{dir_id, rgb_base64}` views → `captions_per_dir` (LIST[TEXT], aligned 1:1) + `captions_json`.

1. Extraction & fidelity

Loader and generate code moved verbatim from navgpt_mp3d_tools._get_blip2 + BLIP2CaptionNode.forward: greedy decode, max_new_tokens=64, prompt prefix "This is a scene of" — the online replica of NavGPT's offline preprocessing (BLIP-2 captions over 24 egocentric views per viewpoint). Because both the code and (by default) the hosting env are unchanged, outputs are unchanged — verified byte-equal 3/3 against the retired in-nodeset chain via captured-input unit replay.

What stayed with the method: the 3-elevation merging + 8-compass direction labelling now lives in the pure node navgpt_mp3d_tools__format_captions, and the numpy→base64 adaptation in navgpt_mp3d_tools__views_to_base64. The wire chain replacing the old fat node is views_to_base64 → model_blip2__caption → format_captions.

2. Canvas Nodes

`model_blip2__caption`

Field	Detail
Inputs	`views` (ANY — ordered list of `{dir_id, rgb_base64}` dicts, e.g. 24 views)
Outputs	`captions_per_dir` (LIST[TEXT] — aligned 1:1 with `views`), `captions_json` (TEXT — same list as JSON)
Config	`model_name` (default `Salesforce/blip2-flan-t5-xl`), `prompt` (default `"This is a scene of"`), `max_new_tokens` (slider 16–256, default 64). Device moved to the deployment level: `$BLIP2_DEVICE`
Backend call	`processor(images=pil, text=prompt)` → greedy `model.generate(max_new_tokens)` → decode; fp16 on CUDA

Stateless — the former per-server caption cache was removed in the 2026-07-05 FM-template alignment: the server holds loaded weights only, and reuse (e.g. repeated identical frames) is a graph-level decision, per the prototype ruling. Identical inputs still produce identical captions (greedy decode); they are just recomputed.

3. Server Mode

class Blip2NodeSet(BaseNodeSet):
    name = "model_blip2"
    parallelism = "shared"  # one server; K eval workers coalesce onto it
    server_python = conda_env_python("ac-fm", "BLIP2_PYTHON")

Runs as its own server subprocess with its own CUDA context (~4 GB fp16). The default env is the shared ac-fm FM env (torch 2.8.0+cu126 + transformers 5.13.0) since 2026-07-05 — captions verified byte-identical to the previous agentcanvas hosting in a greedy-decode parity replay. Point $BLIP2_PYTHON elsewhere to override.

Engines live in a lazy registry keyed by model_name (double-checked threading.Lock; the pre-2026-07-05 module singleton silently ignored a changed model id), with a load-failure latch and a single-flight GPU inference lock; the forward runs in a thread-pool executor so the server's event loop stays responsive. Load: POST /api/components/nodesets/model_blip2/load?mode=server.

4. Environment Variables

Variable	Default	Purpose
`BLIP2_PYTHON`	`ac-fm` env python	Interpreter for the server subprocess (dedicated-env override)
`BLIP2_DEVICE`	`auto` (cuda when available)	Inference device (moved off the node UI 2026-07-05 — deployment knob)

5. Failure Behaviour

Degraded latch — a model-load failure latches (no retry storm); every call then returns empty outputs (captions_per_dir = [], captions_json = "") with a degraded self-log entry (house FM convention since 2026-07-05).
Per-view isolation — a caption exception on one view is caught and logged; that slot stays "" and the remaining views proceed, keeping the 1:1 alignment.
Empty views → [] / "[]" without touching the model.

6. Consumers

spatialnav_mp3d.json — init + step captioning chains (*_views2b64 → model_blip2__caption → *_fmt_caps)
NavGPT — online-captioning alternate that replicates the paper's offline BLIP-2 pass at runtime (not part of the paper-faithful cached path)

See also: model_instructblip (the DiscussNav-prompt sibling captioner) in the Foundation Models index.