Qwen2.5-VL NodeSet
The Qwen2.5-VL NodeSet (VLMQwen25VLNodeSet, workspace/nodesets/model/vlm_qwen2_5_vl.py) wraps Qwen2.5-VL-3B-Instruct as a generic foundation-model nodeset. It serves the ReAct reasoning + VQA VLM behind the ToolEQA method nodeset: the contract mirrors upstream QwenEngine.__call__, so ToolEQA's ReactCodeAgent uses this node as a drop-in llm_engine over the standard server-mode HTTP route.
env: ac-fm
| Primitive | Purpose |
|---|---|
vlm_qwen2_5_vl__generate |
Chat generation: (messages | prompt, image_paths, stop_sequences) → text — image file paths attached to the last user turn, manual stop-sequence truncation. |
1. Canvas Nodes
vlm_qwen2_5_vl__generate
| Field | Detail |
|---|---|
| Inputs | messages (ANY — chat list [{role, content}], preferred; also accepts a JSON string), prompt (TEXT — single-turn fallback), image_paths (ANY — file paths on shared disk), stop_sequences (ANY) |
| Outputs | text (TEXT — post stop-sequence truncation) |
| Config | model_dir (blank = $QWENVL_MODEL_DIR or the repo-anchored 3B default), max_new_tokens (2048), temperature (0.7), top_p (0.8), top_k (100), repetition_penalty (1.05); do_sample follows temperature > 0. All on the node UI since 2026-07-05 (previously a non-rendered config_schema) |
| Backend call | apply_chat_template → qwen_vl_utils.process_vision_info → model.generate → decode of the new tokens only |
Image handling — verbatim port of upstream QwenEngine.call_vlm: images are injected as blocks into the last user turn (prior turns stay text-only); the ReAct agent saves frames to disk and passes paths, which is why the port carries paths rather than base64.
Serialised generation — a lazily-created asyncio.Lock allows only one in-flight generate per server. One shared 3B model is not safe for concurrent torch.generate (KV-cache/activation memory balloons and can CUDA-OOM under K eval workers); a single in-flight call bounds peak VRAM while the workers' habitat/TSDF sides still parallelise.
2. Server Mode
parallelism = "shared" (stateless across calls — K callers coalesce through one hosted copy). server_python defaults to the shared ac-fm FM env (Python 3.11, torch 2.8.0+cu126, transformers 5.13.0, qwen-vl-utils) since 2026-07-05 — greedy generations verified byte-identical to the retired ac-qwenvl hosting under matched sdpa attention. initialize() warms the weights eagerly; they live until subprocess teardown.
Load specifics: bfloat16 with device_map="cuda:0" (the index is mandatory — the allocator warmup rejects a bare "cuda"), flash-attention-2 when the wheel imports with an sdpa fallback (the prebuilt wheel needs glibc ≥ 2.32 — on older hosts the fallback engages; in a newer-glibc Docker base flash-attention re-activates). The 3B checkpoint is a single-3090 budget choice (co-hosts with DetAny3D + Habitat); point $QWENVL_MODEL_DIR at a 7B checkout on a bigger GPU.
3. Environment Variables
| Variable | Default | Purpose |
|---|---|---|
QWENVL_PYTHON |
ac-fm env python |
Interpreter for the server subprocess |
QWENVL_MODEL_DIR |
data/qwen2_5_vl/Qwen2.5-VL-3B-Instruct |
Checkpoint directory (swap for 7B on a bigger GPU) |
4. Degraded Mode
Import or load failure leaves the bundle unset and every generate returns "" (logged) — the graph keeps running with empty VLM signal. A per-call generate exception likewise returns "" rather than raising.
5. Consumers
- ToolEQA (
tooleqamethod nodeset) — the ReAct loop'sllm_engineplus VQA calls, over server-mode HTTP