InternVL3 NodeSet

The InternVL3 NodeSet (VLMInternVL3NodeSet, workspace/nodesets/model/vlm_internvl3.py) wraps InternVL3 via transformers AutoModelForImageTextToText — the other flagship open vision-language model alongside Qwen3-VL, and a consistently strong one on spatial/grounding and document benchmarks. It exposes the same generic generate primitive (a chat message list plus media paths → text) and, like Qwen3-VL, reasons over video as well as images: pass clip paths and it reasons over the temporal sequence, not just a single frame.

env: ac-fm (shared FM env) · backend: transformers AutoModelForImageTextToText + AutoProcessor · default: OpenGVLab/InternVL3-1B-hf (ungated)

1. One primitive: multimodal generate

The tool takes either a full chat messages list (preferred — the caller builds the turns) or a plain prompt string, plus lists of image and video file paths (or URLs) on shared disk. Media is attached to the most recent user turn as chat content blocks; the processor loads it at template time. Stop sequences are truncated from the generated text. The contract mirrors the Qwen3-VL node so the two flagship open VLMs are interchangeable behind one wire shape.

2. Messages, media paths, stops

image_paths and video_paths accept a JSON list (or a single string). They are attached to the last user turn — a turn already in block form is left untouched, so a fully-built messages list passes through verbatim. All generation knobs (max tokens, temperature, top-p, repetition penalty) are node config; temperature = 0 selects greedy decoding.

3. Model & generation

The engine builds the input via the modern apply_chat_template(tokenize=True, add_generation_prompt=True, return_dict=True) path — the processor loads images and video from path/URL directly, so no qwen_vl_utils and no trust_remote_code are needed. Use the transformers-native -hf checkpoints (OpenGVLab/InternVL3-*-hf); the plain (non--hf) repos ship custom code and are intentionally not used here. Generation is greedy or sampled per config, then the prompt tokens are trimmed off before decoding:

msgs = [{"role": "user", "content": [{"type": "image", "image": image_path},
                                     {"type": "video", "video": clip_path},
                                     {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt").to(model.device)
gen = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
text = processor.batch_decode(trimmed, skip_special_tokens=True)[0]

Each model_id is a lazy singleton in a registry (checkpoints coexist). On CUDA it loads in bfloat16 with flash-attn when the wheel is present (sdpa fallback otherwise); on CPU it loads in float32. Generation is single-flight per engine (one in-flight generate bounds peak VRAM under K eval workers — concurrent KV-cache growth can CUDA-OOM).

4. Output

The text port is the decoded generation with the input prompt trimmed and any configured stop sequences cut. Empty on load failure or error (§8).

5. Canvas Node

vlm_internvl3__generate

6. Server Mode

parallelism = "shared" (K callers coalesce through one hosted copy; no per-call state); server_python = conda_env_python("ac-fm", "INTERNVL3_PYTHON"). InternVL3 runs in the shared ac-fm env — the transformers InternVL stack (Python 3.11 + torch 2.8.0+cu126 + transformers 5.13.0) is native there, loaded via AutoModelForImageTextToText + AutoProcessor. Weights are HF-downloaded and load lazily on the first generate. Point model_id at a 2B / 8B / 14B-hf checkpoint on a bigger GPU. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/vlm_internvl3/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm), and any generation exception is caught; both yield "" on the text port with a degraded/error self-log. An empty string is a clean "no answer this step" signal; consumers keep their own fallback and never receive a fabricated reply.

Verified 2026-07-08: OpenGVLab/InternVL3-1B-hf loads as InternVLForConditionalGeneration / InternVLProcessor; an in-process forward() on CPU generated text over a real image (answered a one-word query).