SmolVLM2 NodeSet

The SmolVLM2 NodeSet (VLMSmolVLM2NodeSet, workspace/nodesets/model/vlm_smolvlm2.py) wraps Hugging Face's small, efficient VLM via transformers AutoModelForImageTextToText — the lightweight member of the VLM palette. It comes in 256M / 500M / 2.2B sizes and still reasons over images and video, which makes it the one to reach for when a flagship is overkill or VRAM is tight: edge rollouts, dense per-frame captioning. It exposes the same generic generate primitive as the other VLM nodesets (a chat message list plus media paths → text).

env: ac-fm (shared FM env) · backend: transformers AutoModelForImageTextToText + AutoProcessor · default: HuggingFaceTB/SmolVLM2-256M-Video-Instruct (ungated)

1. One primitive: multimodal generate

The tool takes either a full chat messages list (preferred — the caller builds the turns) or a plain prompt string, plus lists of image and video file paths (or URLs) on shared disk. Media is attached to the most recent user turn as chat content blocks; the processor loads it at template time. Stop sequences are truncated from the generated text. The contract mirrors the other VLM nodesets, so SmolVLM2 slots in wherever a lighter model is enough.

2. Messages, media paths, stops

image_paths and video_paths accept a JSON list (or a single string). They are attached to the last user turn — a turn already in block form is left untouched, so a fully-built messages list passes through verbatim. All generation knobs (max tokens, temperature, top-p, repetition penalty) are node config; temperature = 0 selects greedy decoding.

3. Model & generation

The engine builds the input via the modern apply_chat_template(tokenize=True, return_dict=True) path — the processor loads images and video from path/URL directly. Generation is greedy or sampled per config, then the prompt tokens are trimmed off before decoding:

msgs = [{"role": "user", "content": [{"type": "image", "image": image_path},
                                     {"type": "video", "video": clip_path},
                                     {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True,
                                       return_dict=True, return_tensors="pt").to(model.device)
gen = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
text = processor.batch_decode(trimmed, skip_special_tokens=True)[0]

Each model_id is a lazy singleton in a registry. On CUDA it loads in bfloat16 with flash-attn when the wheel is present (sdpa fallback otherwise); on CPU it loads in float32. Generation is single-flight per engine (concurrent KV-cache growth can CUDA-OOM under K eval workers).

4. Output

The text port is the decoded generation with the input prompt trimmed and any configured stop sequences cut. Empty on load failure or error (§8).

5. Canvas Node

vlm_smolvlm2__generate

6. Server Mode

parallelism = "shared" (K callers coalesce through one hosted copy; no per-call state); server_python = conda_env_python("ac-fm", "SMOLVLM2_PYTHON"). SmolVLM2 runs in the shared ac-fm env (Python 3.11 + torch 2.8.0+cu126 + transformers 5.13.0) — the transformers SmolVLM stack is native there and num2words is installed. Weights are HF-downloaded and load lazily on the first generate. The default 256M-Video checkpoint co-hosts trivially; point model_id at the 500M / 2.2B checkpoints for quality. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/vlm_smolvlm2/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm; a missing num2words is the common cause), and any generation exception is caught; both yield "" on the text port with a degraded/error self-log. An empty string is a clean "no answer this step" signal; consumers keep their own fallback and never receive a fabricated reply.