SpatialBot NodeSet — AgentCanvas

The SpatialBot NodeSet (SpatialBotNodeSet, workspace/nodesets/model/vlm_spatialbot.py) wraps SpatialBot-3B (24.06) — a Bunny-Phi VLM finetuned for RGB-D spatial understanding — as a generic depth-aware VLM foundation-model nodeset. It was extracted from opennav_perception — where SpatialBot lived co-hosted with RAM in one method-owned PerceptionEngine — in the TODO #56 boundary pass (2026-07-04).

env: ac-ram

Primitive	Purpose
`vlm_spatialbot__caption_views`	Keyed-batch captioning: `{key: {rgb_base64, depth_base64?, depth_raw_base64?}}` → `{key: caption}`, keys opaque, insertion order preserved.
`vlm_spatialbot__generate`	Single-view generation: `(rgb_b64, depth_b64?, prompt) → text`.

1. Extraction & fidelity

The model-side semantics moved here byte-faithful to upstream (Open-Nav api.py; the deviations fixed 2026-07-03 are preserved). Four load-bearing details:

(a) Depth packing. Upstream packs the per-tile min-max-normalised uint8 depth image — not raw 16-bit millimetres — so the R channel is degenerate-zero. The explicit int16 promotion keeps NumPy ≥2 (NEP 50) byte-identical to NumPy 1.x's silent promotion:

img = depth_u8.astype(np.int16)  # explicit promotion: NumPy >= 2 (NEP 50) safe
three_channel_array[:, :, 0] = (img // 1024) * 4  # always 0 for uint8 input
three_channel_array[:, :, 1] = (img // 32) * 8
three_channel_array[:, :, 2] = (img % 32) * 8

(b) Image-token splice. Bunny-Phi expects the two <image 1>/<image 2> placeholders spliced into the input ids as special token ids -201 / -202 — passing the literal placeholder text means the model never attends to the images ("I cannot see any images provided").

(c) Generation args. Mirrors api.py:271-277 exactly — upstream relies on the checkpoint's own generation defaults, so injecting do_sample/temperature would diverge:

output_ids = self.model.generate(
    text_ids,
    images=image_tensors,
    max_new_tokens=200,      # api.py:133
    use_cache=True,
    repetition_penalty=1.0,  # nothing else: the checkpoint's own defaults
)

(d) SigLIP vision-tower force-load. The tower is lazily constructed, so the model-level .to(device) misses it; without an explicit load_model() + move it loads onto CPU during generate and raises a device-mismatch 500.

What stayed with the methods: the compute-saving candidate filter. opennav__select_candidate_views / threestepnav__select_candidate_views are pure method nodes that produce the keyed views dict this nodeset consumes.

2. Canvas Nodes

`vlm_spatialbot__caption_views`

Field	Detail
Inputs	`views` (ANY — `{key: {rgb_base64, depth_base64?, depth_raw_base64?}}` keyed dict)
Outputs	`captions` (ANY — `{key: caption}`, every input key covered)
Config	`model_path` (blank = `$SPATIALBOT_PATH` or repo default), `prompt` (default: the spatial what-objects-how-far prompt), `max_new_tokens` (default 200)
Backend call	per view: decode RGB (+ depth, see below) → Bunny-Phi chat template → `generate` per §1(c)

`vlm_spatialbot__generate`

Field	Detail
Inputs	`rgb_b64` (TEXT), `depth_b64` (TEXT, optional — normalised-u8 depth), `prompt` (TEXT)
Outputs	`text` (TEXT)
Config	`model_path` (blank = `$SPATIALBOT_PATH` or repo default), `max_new_tokens` (default 200)

Depth input selection — per view, the packer receives the image upstream actually packs: prefer depth_base64 (the env's per-tile min-max-normalised uint8 encode, byte-identical to upstream generate_input's depth_img); else reconstruct that normalisation from depth_raw_base64 (16-bit absolute depth); with no depth at all, the RGB image is duplicated as the second image (upstream behaviour).

3. Server Mode

parallelism = "shared" (stateless VLM — one server, K eval workers coalesce); server_python = conda_env_python("ac-ram", "SPATIALBOT_PYTHON").

The ac-ram env (transformers 4.39.3 + torch 2.4.1) is in Bunny-Phi's compatible range and is the env that already hosted this exact forward inside opennav_perception — so the extraction cannot shift outputs (verified 10/10 byte-equal via captured-input unit replay). A dedicated env remains the ideal; point $SPATIALBOT_PYTHON at it when built.

Engines live in a lazy registry keyed by model_path (fp16, trust_remote_code), with a load-failure latch and a single-flight GPU inference lock (FM template, 2026-07-05). Checkpoint path resolution walks upward from the nodeset file to the repo's data/ anchor — a fixed ../../.. breaks when the file is served from a workspace overlay copy at a different depth. Load: POST /api/components/nodesets/vlm_spatialbot/load?mode=server.

4. Environment Variables

Variable	Default	Purpose
`SPATIALBOT_PYTHON`	`ac-ram` env python	Interpreter for the server subprocess
`SPATIALBOT_PATH`	`data/opennav/SpatialBot-3B`	Checkpoint directory
`OPENNAV_SPATIALBOT_PATH`	—	Legacy alias from the `opennav_perception` era (still honoured)

initialize(spatialbot_path=…) also accepts the checkpoint path as a nodeset kwarg.

5. Failure Behaviour

Degraded latch — a model-load failure latches; every call then returns empty outputs with a degraded self-log entry (house FM convention since 2026-07-05).
A view missing rgb_base64 yields "" for that key; the keyed output always covers every input key.
Empty views / missing rgb_b64 → empty output without touching the model.

6. Consumers

Three-Step Nav — threestepnav_ce.json (verified): threestepnav__select_candidate_views → vlm_spatialbot__caption_views feeding the navigator's observation formatter.
Open-Nav — opennav_habitat.json (unverified): same shape via opennav__select_candidate_views.

See also: model_ram (the RAM/RAM++ tagger it used to be co-hosted with) in the Foundation Models index.