Qwen3-VL NodeSet

The Qwen3-VL NodeSet (VLMQwen3VLNodeSet, workspace/nodesets/model/vlm_qwen3_vl.py) wraps Qwen3-VL via transformers Qwen3VLForConditionalGeneration — the successor to the Qwen2.5-VL nodeset. It exposes the same generic generate primitive (a chat message list plus media paths → text), but Qwen3-VL is newer and stronger and, crucially, reasons over video as well as images: pass clip paths and it reasons over the temporal sequence — VQA over a rollout, "what changed", temporal grounding — not just a single frame.

env: ac-fm (shared FM env) · backend: transformers Qwen3VLForConditionalGeneration + AutoProcessor · default: Qwen/Qwen3-VL-2B-Instruct (ungated)

1. One primitive: multimodal generate

The tool takes either a full chat messages list (preferred — the caller builds the turns) or a plain prompt string, plus lists of image and video file paths (or URLs) on shared disk. Media is attached to the most recent user turn as Qwen content blocks; the processor loads it at template time. Stop sequences are truncated from the generated text. The contract mirrors the Qwen2.5-VL node so it is a drop-in upgrade, with video_paths as the new port.

2. Messages, media paths, stops

image_paths and video_paths accept a JSON list (or a single string). They are attached to the last user turn — a turn already in block form is left untouched, so a fully-built messages list passes through verbatim. All generation knobs (max tokens, temperature, top-p/k, repetition penalty) are node config; temperature = 0 selects greedy decoding.

3. Model & generation

The engine builds the input via the modern apply_chat_template(tokenize=True, return_dict=True) path — the processor loads images and video from path/URL directly, so no qwen_vl_utils dependency is needed. Generation is greedy or sampled per config, then the prompt tokens are trimmed off before decoding:

msgs = [{"role": "user", "content": [{"type": "image", "image": image_path},
                                     {"type": "video", "video": clip_path},
                                     {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt").to(model.device)
gen = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
text = processor.batch_decode(trimmed, skip_special_tokens=True)[0]

Each model_id is a lazy singleton in a registry. On CUDA it loads in bfloat16 with flash-attn when the wheel is present (sdpa fallback otherwise); on CPU it loads in float32. Generation is single-flight per engine (concurrent KV-cache growth can CUDA-OOM under K eval workers).

4. Output

The text port is the decoded generation with the input prompt trimmed and any configured stop sequences cut. Empty on load failure or error (§8).

5. Canvas Node

vlm_qwen3_vl__generate

6. Server Mode

parallelism = "shared" (K callers coalesce through one hosted copy; no per-call state); server_python = conda_env_python("ac-fm", "QWEN3VL_PYTHON"). Qwen3-VL runs in the shared ac-fm env — Qwen3VLForConditionalGeneration is native to the transformers stack there. Weights are HF-downloaded and load lazily on the first generate. Point model_id at a 4B / 8B / 32B checkpoint on a bigger GPU. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/vlm_qwen3_vl/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm), and any generation exception is caught; both yield "" on the text port with a degraded/error self-log. An empty string is a clean "no answer this step" signal; consumers keep their own fallback and never receive a fabricated reply.