Florence-2 NodeSet

The Florence-2 NodeSet (Florence2NodeSet, workspace/nodesets/model/model_florence2.py) wraps Microsoft's Florence-2 unified vision model via transformers Florence2ForConditionalGeneration. Florence-2 is a single compact seq2seq model that does many perception tasks through one interface: you pass a task token and it emits the answer — a caption, a set of detection boxes, region-grounded phrases, or OCR. One model covers ground that would otherwise take a captioner + a detector + a grounder + an OCR engine, which is exactly what an agent's "look at this frame and tell me X" step wants: switch the task, reuse the weights.

env: ac-fm (shared FM env) · backend: transformers Florence2ForConditionalGeneration + AutoProcessor · default: florence-community/Florence-2-base (ungated)

1. One primitive, many tasks

The tool exposes a single node whose behaviour is chosen by the task config. The three tasks marked (text) also consume the text_input config — a phrase or category to ground; the rest ignore it.

2. Image in — one at a time

The image port accepts a single {rgb_base64} dict or a raw base64 string (a one-element list is unwrapped). Florence-2 generates per image and its result is a per-image dict, so this node is deliberately single-image; loop at graph level for a batch. A missing or malformed image degrades to "".

3. Model & generation

The prompt is the task token, optionally followed by text_input for the grounding / open-vocab / referring tasks. Florence-2 generates the answer as a tagged token string, which the processor's post_process_generation parses into a structured dict keyed by the task token:

prompt = task + text_input                    # e.g. "<CAPTION_TO_PHRASE_GROUNDING>a chair"
inputs = processor(text=prompt, images=image, return_tensors="pt")
gen_ids = model.generate(input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"],
                         max_new_tokens=1024, num_beams=3, do_sample=False)
text = processor.batch_decode(gen_ids, skip_special_tokens=False)[0]
result = processor.post_process_generation(text, task=task, image_size=(W, H))

Boxes and polygons come back in original pixel coordinates — post_process_generation rescales from Florence-2's internal 1000×1000 grid to the image's (W, H). Each model_id is a lazy singleton in a registry, frozen and moved to the device once (fp16 on CUDA, fp32 on CPU); inference is single-flight per engine.

4. Result envelope

The output result is post_process_generation's dict serialized as JSON, keyed by the task token. For a caption:

{"<CAPTION>": "a yellow square on a black background"}

For object detection (boxes are [x1, y1, x2, y2] in pixels):

{"<OD>": {"bboxes": [[64, 40, 210, 300]], "labels": ["chair"]}}

Referring / region segmentation tasks return polygons instead of bboxes; OCR-with-region returns quad_boxes. The dict is returned verbatim so downstream nodes parse the shape they expect for the task they asked for.

5. Canvas Node

model_florence2__run

6. Server Mode

parallelism = "shared" (a stateless generator — one server across eval workers); server_python = conda_env_python("ac-fm", "FLORENCE2_PYTHON"). Florence-2 runs in the shared ac-fm env — Florence2ForConditionalGeneration is native to the transformers stack there. Use a florence-community/* checkpoint: the original microsoft/Florence-2-* repos still ship the remote-code layout and do not load natively (missing lm_head + a tokenizer with no image_token). The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_florence2/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, a missing/malformed image, or an unknown task token yields "" on the node's output. An empty result is a clean "no answer this step" signal; consumers keep their own fallback and never receive a fabricated detection or caption.