VoxPoser on AgentCanvas — paper analysis + voxposer port

VoxPoser (Huang et al. 2023; repo huangwl18/VoxPoser) is a zero-shot manipulation method: a chain of Language Model Programs (LMPs) writes Python that composes 3-D value maps (affordance / avoidance / rotation / velocity / gripper) over a voxel grid, a greedy planner extracts a waypoint path, and the robot follows it. This page documents the decomposed, env-decoupled port — the 8-node local voxposer nodeset planning each subtask on a frozen GT scene snapshot and following waypoints closed-loop on LIBERO (Franka) via env_libero__step_ee_pose in voxposer_libero_decomposed.json. Upstream's RLBench env is replaced wholesale; this is a cross-simulator port, not a reproduction on the paper's benchmark.

1. Upstream method analysis (Huang et al. 2023)

Upstream runs everything in-process against a live RLBench env. Per instruction:

1. Planner LMP (prompts/rlbench/planner_prompt.txt): the instruction becomes Python that calls composer("<subtask>") once per subtask, in order. LMP config: gpt-4 · temp 0 · max_tokens 512 · stop ["# Query: ", "objects = "] (rlbench_config.yaml).
2. Composer LMP per subtask (composer_prompt.txt, include_context: False): writes code that calls parse_query_obj (entity grounding) and the five get_*_map LMPs, then execute(movable, affordance_map=…, …).
3. Value maps: each get_*_map LMP returns a callable evaluated lazily; unspecified maps fall back to current-state defaults (interfaces.py). Maps live on a map_size=100 voxel grid over the workspace bounds.
4. Plan (execute() first half, interfaces.py:169–): object_centric = movable ∉ EE_ALIAS (:183); the greedy voxel planner (planners.py — gradient-free nearest-high-value descent with obstacle weighting, Savitzky–Golay smoothing, curvature cut) optimizes a path from the movable's position; postprocessing yields (position, rotation, velocity, gripper) waypoints.
5. Follow (execute() second half): per waypoint, controller.execute(movable_obs, waypoint) (:162) — for the EE this reduces to a motion-planner move; for object-centric subtasks (pushing) it is a true MPC step: sample num_samples=10000 contact/push candidates through a heuristic PushingDynamicsModel, take the best, re-observe, repeat. After the loop, a final apply_action snaps to the last waypoint (:184–205, EE-centric only).
6. Replan: max_plan_iter outer repetitions of plan→follow per subtask (config default 1 — effectively open-loop per subtask in sim).
7. LLM caching: every API call goes through a pickle DiskCache keyed by the full request kwargs (LLM_cache.py) — re-runs are free.

1.1 Upstream invariants worth marking

• The LLM writes code, the code runs. Every LMP exec()s generated Python in a namespace exposing detect / execute / get_*_map / numpy. The method's behaviour is the prompt files plus this namespace — nothing else.
• Planning is over value maps, not poses. The five map LMPs return callables; the planner re-evaluates them lazily so maps always reflect the latest observation.
• object_centric selects the follow mechanism (interfaces.py:183): EE subtasks get motion-planner moves; pushing subtasks get true MPC through PushingDynamicsModel. The planned path is the movable's path either way — for pushing, waypoints are object positions, not EE positions.
• max_plan_iter: 1 in the sim config — one plan per subtask, with num_waypoints_per_plan: 10000 commented "we only do open loop for sim". Replanning is a real-robot affordance.
• The disk cache makes runs reproducible — identical prompts at temp 0 replay from pickle, no API call.

2. The AgentCanvas port

The port is a plan/follow split across a two-scope graph, with the planning stack fully env-decoupled (commit a90cc7dc, 2026-06-11). The vendored LMP stack runs local-mode in the backend process against a WiredEnv — a read-only env stand-in backed by a frozen GT snapshot from env_libero__observe_objects. This is sound because planning never steps the sim: every env read during one plan_subtask sees a single static state, so a snapshot is semantically identical to the live reads the retired in-subprocess adapter performed. Only two sim-driving entry points are stubbed in the composer's namespace: execute → LMP_interface.plan (plan, don't drive) and reset_to_default_pose → a single lift-to-home waypoint (z = 1.20). WiredEnv raises on any write call — the loud-failure guard motivated by the abandoned v2 design, which buffered execute() open-loop and scored SR 0.

2.1 Per-episode flow

Drawn simplified in three places. ① vp_planner and vp_plan_subtask are violet-amber hybrids: the LLM call happens inside the vendored LMP stack (litellm + disk cache), not on a canvas wire — there is no llmCall node in this graph. ② The outer pivot pair carries subtask_index / episode_success / step_index (persist) plus init ports for runtime_handle / subtask_list / subtask_count / max_steps; obs_loop re-fires every outer iteration (triggered by iter_in_outer.init_subtask_list) so each subtask plans against a fresh snapshot. ③ Termination: vp_check_done.done → iter_out_outer.stop (success, all-subtasks-complete, or env step budget), and post-loop iter_out_outer.final_stop → evaluate.trigger — the sound pivot-rooted shape. The inner follow loop lives inside the waypoint_exec composite (its own iterIn/iterOut, budget 80) and drives every waypoint closed-loop through step_pose; hold_steps > 0 entries tell the env to skip OSC convergence and run N zero-delta physics ticks with the new gripper command.

2.2 Node inventory

2.3 State & memory

2.4 Boundary contract

Method side runs local-mode (no server_python); LIBERO stays in its env subprocess. The only env coupling is the read side — env_libero__observe_objects, one privileged GT snapshot dict (BDDL object names, per-object AABB point clouds, scene PC, EE pose, gripper, workspace bounds) — and the write side, env_libero__step_ee_pose waypoint actions. Any env exposing that observe/step pair can host VoxPoser. Boundary deviation worth knowing: the LLM is not consumed via a foundation-model nodeset — the vendored LMP._cached_api_call calls litellm directly (recorded in the LMP.py header: cache-shape preservation, retries delegated, chat-only). The TODO #56 method/FM split therefore does not apply to this nodeset's LLM usage.

2.5 Prompt assets

workspace/nodesets/method/voxposer/prompts/libero/ — 8 files matching upstream's prompts/rlbench/ names. Byte-diff this audit: 7 of 8 are byte-identical to the RLBench originals (modulo trailing newline) — planner, composer, and all five get_*_map prompts carry no LIBERO adaptation at all. Only parse_query_obj_prompt.txt was rewritten: the few-shot examples now use LIBERO BDDL object vocabulary (alphabet_soup_1, akita_black_bowl_1, …) so entity grounding sees in-domain names. LMP configs (model / 512 / temp 0 / stop / include_context: False for the composer / has_return for the map LMPs) reproduce rlbench_config.yaml.

3. Delta vs upstream — four buckets

A. Preserved verbatim

B. Forced by substrate (script → typed graph)

C. Forced by environment / cost

D. Intentional divergences

E. Unexplained / defects

Probed and cleared: stop-token nuance ("objects =" vs upstream planner's "objects = " — the no-space form is a prefix, so generation stops at the same boundary); all 12 planner config keys byte-equal to yaml; dynamics_models.py byte-identical; _fill_default_voxel_maps refactor branch-equivalent to upstream's inline defaults; empty-path EE fallback (plan() emits one current-pose waypoint where upstream relied on the in-execute final apply_action — inline-documented); the LIBERO single-execute-per-subtask idiom (multiple execute calls concatenate, documented in plan_subtask). No equivalence test exists for this nodeset — the v1-monolith ground truth was deleted with the monolithic graph (per user decision, commit a90cc7dc), so eval parity (§4) is the only executable check.

4. Evaluation

Current status: voxposer_libero_decomposed.json sits in workspace/graphs/vla/unverified/ — moved 2026-06-10 with the env_libero gym migration + decoupling, pending a clean re-validation run. The 3/5 reference predates the snapshot transport; the decoupling argument (frozen snapshot ≡ live reads, because planning never steps the sim) says behaviour should be unchanged, but that claim is exactly what the pending re-eval must confirm. Note the LLM disk cache when interpreting re-runs: with load_cache=true and temp 0, repeated runs replay cached plans — a change in outcome between cached runs implicates the follow/env side, not the LMPs.

5. Usage

# load — method is local-mode; env is server-mode
POST /api/components/nodesets/voxposer/load
POST /api/components/nodesets/env_libero/load?mode=server

# graph
workspace/graphs/vla/unverified/voxposer_libero_decomposed.json

# batch eval — graph-only form: /experiment:run <profile> <graph_name> [key=value ...]
/experiment:run smoke_voxposer_libero_decomposed voxposer_libero_decomposed episode_count=1
/experiment:run perf_voxposer_libero_decomposed voxposer_libero_decomposed

6. What this nodeset is NOT

• Not the paper's benchmark. Upstream is RLBench/real-robot; this is a LIBERO cross-simulator port. There is no "paper number" to close against — the port is benchmarked against its own locked runs.
• Not a SIMPLER method. VoxPoser in this project runs on LIBERO + Franka; SIMPLER is reserved for the native RT-1-X / Octo baselines.
• Not an llmCall consumer. The LLM rides inside the vendored LMP stack (litellm + disk cache), not a foundation-model nodeset (§2.4).
• Not perception-driven. Planning reads a privileged GT snapshot (BDDL names + point clouds), faithful to upstream's use of sim ground truth — there is no detector in the loop.
• Not env logic. OSC waypoint following, convergence, hold-tick execution and success scoring live in env_libero.

7. Source files