AgentCanvas / Pages / Developer Guide / Nodesets / Model / InstructBLIP NodeSet
2026-07-05

InstructBLIP NodeSet

The InstructBLIP NodeSet (InstructBlipNodeSet, workspace/nodesets/model/model_instructblip.py) wraps InstructBLIP (23.05) FlanT5-XL scene captioning as a generic foundation-model nodeset. Extracted from navgpt_mp3d_tools per the method/FM boundary (TODO #56): per-view scene description is a generic vision primitive, consumed by DiscussNav and NavGPT-MP3D alike — exactly like model_ram for tagging.

env: ac-fm

Primitive Purpose
model_instructblip__caption Per-direction scene description: ordered list of {dir_id, rgb_base64} → captions_per_dir (LIST[TEXT], aligned 1:1) + captions_json.

1. Fidelity

The DiscussNav recipe: LAVIS blip2_t5_instruct/flant5xl with the prompt "Describe this indoor scene in details", one caption per direction, later merged as Scene Description: {instructblip} Scene Objects: {ram};. This nodeset uses the HF transformers equivalent (LAVIS isn't installed) with decoding aligned to LAVIS Blip2T5Instruct.generate defaults — beam-5, max_length 256, repetition_penalty 1.5, length_penalty 1.0, do_sample=False. LAVIS calls the same HF T5 .generate under the hood, so matching kwargs makes output equivalent up to float noise.

Manual image-token unpacking — InstructBlipProcessor.__call__ concatenates the image-token list with the text tensor and trips a "list + Tensor" TypeError; the node bypasses it by calling the sub-tokenizers directly and prepending processor.image_token.content × num_query_tokens itself. This relies on processor attributes added around transformers 4.44 — one reason the hosting env matters (§3).

2. Canvas Nodes

model_instructblip__caption

Field Detail
Inputs views (ANY — ordered list of {dir_id, rgb_base64}, e.g. 12 directions; same tiles that feed model_ram)
Outputs captions_per_dir (LIST[TEXT] — aligned 1:1), captions_json (TEXT)
Config model_name (default Salesforce/instructblip-flan-t5-xl), prompt, max_length (slider 32–384, default 256), num_beams (slider 1–8, default 5). Device moved to the deployment level: $INSTRUCTBLIP_DEVICE

Stateless — the former per-server caption cache was removed in the 2026-07-05 FM-template alignment. That removal also killed a real bug: the cache was keyed on image bytes only, so a prompt/beam/model config change within one server lifetime kept serving captions generated under the old config. The server now holds loaded weights only; reuse is a graph-level decision. After each batch the node still calls torch.cuda.empty_cache() to curb allocator growth.

3. Server Mode

parallelism = "shared" (one ~7 GB model shared by K eval workers; bit-identical at worker_count=1). server_python defaults to the shared ac-fm FM env (transformers 5.13.0 + tokenizers 0.22) since 2026-07-05 — beam-5 captions verified byte-identical to the previous agentcanvas hosting, manual image-token unpacking included. Not ac-ram: its tokenizers 0.15.2 cannot parse the flan-t5 fast tokenizer and it lacks sentencepiece, so InstructBlipProcessor fails to load there (a past mis-choice that 500-ed every caption call). Override with $INSTRUCTBLIP_PYTHON. Engines live in a lazy registry keyed by model_name with a load-failure latch and a single-flight GPU inference lock (FM template, 2026-07-05). Load: POST /api/components/nodesets/model_instructblip/load?mode=server.

4. Environment Variables

Variable Default Purpose
INSTRUCTBLIP_PYTHON ac-fm env python Interpreter for the server subprocess (dedicated-env override)
INSTRUCTBLIP_DEVICE auto (cuda when available) Inference device (moved off the node UI 2026-07-05 — deployment knob)

5. Failure Behaviour

Per-view caption exceptions are caught and logged; that slot stays "" and the remaining views proceed (1:1 alignment preserved). A model-load failure latches and every call returns empty outputs with a degraded self-log entry (house FM convention since 2026-07-05). Empty views → empty outputs without touching the model.

6. Consumers

AgentCanvas docs