MIP ← index / works
loading…
Run your model on the probe

Harness, prompts and evaluation are in the MIP code. Run the frozen boards — rand100 for R2R-CE, mip100 for the other five — and send the run directory (episode_*.jsonl + summary.json) to get listed.

Interfaces

Every interface the probe has been run with: the tools the model gets, the action set, the budgets and the briefing it receives, verbatim from the experiment files. The Interface column above links here.

loading…
01 — who directs the loop

Policy, workflow, or the model itself

Embodied navigation systems differ in who directs the interaction loop. With a trained end-to-end policy, an external rollout loop feeds frames and the instruction to the model and executes the predicted action, step after step — NaVid and StreamVLN, and at industrial scale Qwen-RobotNav. Many zero-shot systems instead place a frozen model inside a hand-designed workflow — MapGPT, DiscussNav — that expands each model call into a predefined graph of perception, memory, planning and verification; dual-system models such as ABot-N1 and InternVLA-N1 couple a slow reasoner to a trained action expert through a fixed handoff.

With agentic control, the model chooses how the interaction proceeds: from the goal and the feedback so far it decides whether to observe again, act, check a result, revise its plan, or stop, and the harness only executes the calls. As frontier vision–language models improve at multi-turn, multimodal reasoning and spatial understanding, we ask how much of embodied interaction the model can direct through a minimal interface. We call this organization agentic embodied control.

fig 1who directs the loop · the minimal interface · key results
The paper's Figure 1. Left: policy, workflow and agentic control, each with trained or zero-shot examples. Middle: the minimal-interface probe, a frontier model directing the loop through observe() and step() over MCP, with no depth, panorama, pose, map, memory, waypoints, search or navigation training. Right: success of our best agent against the best published trained policy and the human tester on R2R-CE, RxR-CE, HM3D v1 and v2, MP3D and HM-EQA.
◆The paper's Fig. 1. Left: policy, workflow and agentic control (T trained · ZS zero-shot). Middle: the minimal interface. Right: our best agent against the best published trained policy and the human tester — published policies report their full validation splits; ours and the human, the 100-episode subsets.
02 — the probe

One camera. Primitive actions. Nothing else.

The task-specific interface is two tools served over the Model Context Protocol, plus a short briefing that states the instruction, the action table and the budget. Each episode is a fresh agent session in a simple synchronous loop: the model picks the next tool call from the interaction history, the harness executes it and returns the result. Whatever performance remains must come from the model itself.

removedthe usual machinery
  • panoramic observation
  • depth & pose
  • metric / topological map
  • explicit memory module
  • planner & search
  • waypoint candidates
  • collision signal
  • navigation-specific training
remainingthe whole interface
  • observe() → one front-facing RGB frame
  • 512×512, does not advance the environment
  • step(actions) → primitives, in order
  • FORWARD 0.25 m · LEFT 15° · RIGHT 15° · STOP
  • returns actions executed + budget left, no image
  • ObjectNav & EQA → + look up · look down
  • HM-EQA ends with an answer instead of STOP
  • budget per episode
  • 200 model turns · 500 actions · 2400 s

OpenAI models run in Codex CLI, Claude models in the Claude Agent SDK, and Qwen in the open mini-swe-agent — each connected through the same tools; every briefing, verbatim, is in the Interfaces section above. Each benchmark is one fixed 100-episode subset shared by all models: R2R-CE uses the common rand100 val-unseen sample introduced by Open-Nav; the other five use mip100, stratified by scene and selected by a hash of the episode identity, so anyone with the official files can rebuild the same 100 episodes.

03 — watch it run

A real episode, decision by decision

Episode 10 of the R2R-CE board, mini-swe-agent driving fable-5, replayed from its log — every frame is the exact image the agent saw, every quote is its own recorded reasoning, every step() is the exact call it issued. It turns around, fights its way past the bed corner, wanders into the wrong room, rebuilds its mental map out loud, and stops — correctly — in the bathroom. The paper's Fig. 2 shows this same episode.

replay — mini-swe-agent × fable-5 · ep 10 · R2R-CE val-unseen// from episode_10.jsonl
instruction “Turn around and walk to the end of the bed. At the end of the bed turn right and walk to the hallway. Once at the entrance way, walk through the hallway to the next room.” SUCCESS · stops 2.98 m from goal
Agent's monocular camera view for the current observation obs 1 / 20
◆20 observe() · 20 step() · 111 primitives · 332 s. Amber marks the segments the model itself described as blocked or off-route: the bed-corner scrape (5–8) and the wrong-room detour it detects and unwinds (14–15). Use the tape or arrow keys to scrub.
session — fable-5 · R2R-CE ep 10// the same episode as a video · left: agent session · right: what it sees
04 — findings

What the paper finds

R1

A minimal interface supports strong performance across tasks. The leading models — opus-5.5, fable-5.1, gpt-6-astra and gpt-6.1-sol — set the best scores on all six benchmarks. The best agents reach 82–83 SR on R2R-CE against our human tester's 94, 89 on RxR-CE against 83, and 81 on HM-EQA. On the same interface, R2R-CE success still spans 38.0 to 83.0 across models: capability lives in the model.

six benchmarksMinimal interface · 14 models · ranked per benchmark
R2R-CESR0255075100fable-5.1opus-5.5gpt-6-astragpt-6.1-solopus-5gpt-6-solfable-5gpt-5.6qwen3.8-maxsonnet-5haiku-5.5gpt-5.5gpt-6-lunagpt-5.6-luna8382.375.77570.769.768.356.756.351.3494640.338human 94
RxR-CESR0255075100gpt-6-astragpt-6.1-solfable-5.1opus-5.5opus-5fable-5gpt-6-solgpt-5.6sonnet-5qwen3.8-maxgpt-6-lunagpt-5.5haiku-5.5gpt-5.6-luna8983797467626253454342423729human 83
HM3D v1SR0255075100gpt-6-astraopus-5.5fable-5.1gpt-6-solgpt-6.1-solopus-5fable-5sonnet-5gpt-5.6gpt-6-lunagpt-5.5qwen3.8-maxhaiku-5.5gpt-5.6-luna8180787574726966636261616053human 85
HM3D v2SR0255075100gpt-6-astragpt-6.1-solopus-5.5fable-5.1opus-5gpt-6-solfable-5sonnet-5gpt-5.6gpt-6-lunagpt-5.6-lunahaiku-5.5qwen3.8-maxgpt-5.59189888786858477736968686765human 89
MP3DSR0255075100opus-5.5gpt-6-astraopus-5fable-5.1gpt-6.1-solfable-5gpt-6-solhaiku-5.5sonnet-5gpt-5.6qwen3.8-maxgpt-6-lunagpt-5.5gpt-5.6-luna6966636362605553525251464641human 71
HM-EQAAcc0255075100opus-5.5gpt-6-astragpt-6-solgpt-6.1-solfable-5.1opus-5gpt-5.6fable-5gpt-5.6-lunahaiku-5.5sonnet-5gpt-5.5gpt-6-lunaqwen3.8-max8180797979777169676464605852human 87
table view
modelR2R-CERxR-CEHM3D v1HM3D v2MP3DHM-EQA
gpt-5.6-luna382953684167
gpt-6-luna40.34262694658
gpt-5.5464261654660
haiku-5.5493760685364
sonnet-551.34566775264
qwen3.8-max56.34361675152
gpt-5.656.75363735271
fable-568.36269846069
gpt-6-sol69.76275855579
opus-570.76772866377
gpt-6.1-sol758374896279
gpt-6-astra75.78981916680
opus-5.582.37480886981
fable-5.1837978876379
human tester948385897187
◆One panel per benchmark: the fourteen models ranked, bars and names coloured by vendor, the dashed line our human tester on the same episodes. R2R-CE values are three-run means; ObjectNav SR = STOP within 1 m.
R2

Who directs the loop matters. Placed inside the SmartWay and O2C-Nav workflows on the same R2R-CE episodes, the strongest models lose ground: opus-5.5 falls from 82.3 to 65 and 69 SR, fable-5.1 from 83.0 to 62 in both. Only weaker models gain (gpt-5.5: 46.0 → 49 and 54). A stronger base model does not automatically make a stronger navigator inside a fixed workflow.

same model, different directorR2R-CE · SR · 7 models
native agent loop (model directs)SmartWay workflowO2C-Nav workflow020406080100gpt-6-luna40.33841gpt-5.5464954gpt-5.656.74558gpt-6-sol69.75261gpt-6-astra75.75972opus-5.582.36569fable-5.1836262
table view
modelnativeSmartWayO2C-Nav
gpt-6-luna40.33841
gpt-5.5464954
gpt-5.656.74558
gpt-6-sol69.75261
gpt-6-astra75.75972
opus-5.582.36569
fable-5.1836262
◆Three bars per model, same R2R-CE episodes. Blue: the model directs its own loop through the Minimal interface. Gold / magenta: the same model as the reasoning module of the SmartWay / O2C-Nav workflow, which adds a waypoint module and fixes when the model is asked. Weakest model at the top, strongest at the bottom: the workflows roughly match or help the weaker ones (gpt-5.5 +3 / +8) and cost the four strongest 3.7 to 21 points.
R3

A simple harness retains strong performance. Replacing the vendor harness with the lightweight mini-swe-agent moves SR by roughly −8 to +8 across twelve models; for the leaders the change is small (fable-5.1 83.0 → 84, opus-5.5 82.3 → 80, gpt-6-astra 75.7 → 79). Serving and context handling differ across harnesses, so this compares complete systems. Toggle the Harness chips on the R2R-CE tab to see every row.

vendor harness vs mini-swe-agentR2R-CE · SR · 12 models
vendor harness (Codex CLI / Claude Agent SDK)mini-swe-agent30405060708090mini − vendorgpt-6-luna+7.7gpt-5.5+6haiku-5.5−2sonnet-5+1.7opus-4.8+7.3gpt-5.6+3.3fable-5+3.7gpt-6-sol−7.7opus-5−1.7gpt-6-astra+3.3opus-5.5−2.3fable-5.1+1
table view
modelvendor harnessmini-swe-agentΔ
gpt-6-luna40.348+7.7
gpt-5.54652+6.0
haiku-5.54947-2.0
sonnet-551.353+1.7
opus-4.855.763+7.3
gpt-5.656.760+3.3
fable-568.372+3.7
gpt-6-sol69.762-7.7
opus-570.769-1.7
gpt-6-astra75.779+3.3
opus-5.582.380-2.3
fable-5.18384+1.0
◆Blue: the vendor harness; gold: mini-swe-agent; the arrow runs from the first to the second. Same Minimal interface and episodes; only the agent loop around the model changes. Vendor values are three-run means. The arrows stay short for every model — the ranking is the model's, not the harness's.
R4

Waypoints help when optional, not when imposed; SLAM adds on top. Moving only through a trained waypoint predictor rescues weak models (qwen3.5-4b 5 → 43 SR) but changes the four strongest by at most two points. Offered alongside primitive actions (Hybrid), it lifts opus-5.5 from 82.3 to 85.7 with 0.59× the model calls, 0.62× the wall time and 0.88× the cost. Adding ORB-SLAM3 tools (Hybrid + SLAM) reaches 88.0 SR and cuts navigation error from 2.65 to 2.09 m — the highest SR for every strong model tested.

imposed waypointsR2R-CE · SR · 12 models
primitive actions (Minimal)waypoints only0153045607590waypoint − primitiveqwen3.5-4b+38qwen3.5-9b+37qwen3.5-plus+11.7gpt-5.5+21haiku-5.5+10sonnet-5+8.7opus-4.8+9.3gpt-5.6+16.3fable-5+0.7opus-5−0.7opus-5.5+1.4fable-5.1−2
table view
modelprimitivewaypoint onlyΔ
qwen3.5-4b543+38.0
qwen3.5-9b744+37.0
qwen3.5-plus41.353+11.7
gpt-5.54667+21.0
haiku-5.54959+10.0
sonnet-551.360+8.7
opus-4.855.765+9.3
gpt-5.656.773+16.3
fable-568.369+0.7
opus-570.770-0.7
opus-5.582.383.7+1.4
fable-5.18381-2.0
◆Blue: primitive actions only (Minimal). Gold: the agent can move only by picking one of the predictor's waypoints; the arrow runs from the first to the second. The gain shrinks as the model gets stronger and vanishes at the top. gpt-5.6's waypoint run used medium effort.
every interface, five models · R2R-CE SR · darker = higher · best per row outlined
modelMinimalWaypointHybridSLAMHybrid + SLAM
haiku-5.54959515156
fable-568.36977—78
opus-570.770787680
opus-5.582.383.785.782.788
fable-5.18381837986

Minimal values are three-run means; opus-5.5 is three runs in every column. Waypoint: waypoints only. Hybrid: waypoints offered next to primitives. SLAM: ORB-SLAM3 pose, map, trajectory and path-preview tools on top of Minimal or Hybrid — no ground-truth pose or map. fable-5 has no SLAM-only run.

the tools, and what each one returnsreal outputs from one Hybrid + SLAM episode
Minimal
observe()one front RGB frame
observe() output: a single front-facing RGB frame of a red dining room.
step([2,2,1,1,1])primitives, in order
{"executed": 5,
 "requested": 5,
 "steps_taken_total": 247,
 "steps_remaining_approx": 253,
 "episode_over": false}
Waypoint
observe_waypoints()four views + the predictor's numbered candidates
observe_waypoints() output: Left, Front, Right and Back views of the dining room with green numbered circles 1 to 5 marking waypoints.
goto(5)walk to a candidate
{"moved_to": 5,
 "direction": "Left",
 "distance_m": 2,
 "steps_taken_total": 8,
 "moves_remaining": 29}
SLAM
get_map()occupancy map from RGB-D, you in blue
get_map() output: top-down occupancy map, white free space, black obstacles, gray unexplored, the agent's path and heading in blue.
preview_path(-4.2, -1.5)planned route, nothing moves
preview_path() output: the map with the walked path in blue and the planned route in magenta to an X target.
look_around()four headings, no movement
look_around() output: four camera views labelled front, right, back and left.
get_pose()SLAM pose estimate
{"x": -1.716,
 "z": 0.997,
 "yaw_deg": -60.1,
 "slam": "ok"}
◆Every panel is exactly what the tool returned to opus-5.5 in one R2R-CE episode (ep 550). Hybrid = Minimal + Waypoint tools; Hybrid + SLAM adds the SLAM tools — no ground-truth pose or map.
R5

Newer models are faster and cheaper, too. From opus-5 to opus-5.5 the median episode falls from 3.8 min to 66 s, model calls from 41 to 25 and cost from $0.70 to $0.14, while SR rises from 70.7 to 82.3; gpt-6.1-sol finishes in 98 s for $0.11. API instability pushed many Qwen episodes to the 2400 s timeout, which affects both runtime and success.

runtime and costR2R-CE · Minimal · default effort · per-episode medians
Per-episode medians on R2R-CE for Claude, OpenAI and Qwen models: success rate, wall-clock time, model calls, fresh input plus output tokens, and vendor-priced cost. opus-5.5 reaches 82 SR in 66 seconds, 25 calls and 0.14 dollars per episode.
◆The paper's runtime figure: wall-clock, model calls, fresh tokens and vendor-priced cost per episode, failed episodes included; SR beside each model. The bottom model of each line is outlined.
05 — where it fails

Where it still fails

We audited the logs and observations of one opus-5.5 run per benchmark: 126 failures under the current criteria — 44 in VLN-CE, 63 in ObjectNav, 19 in HM-EQA. The review is model-assisted and has not been independently checked by humans.

L1

ObjectNav failures include visible targets. Of 63 failures, 39 are labelled target-visible and one has a disputed category; the rest are five wrong targets, sixteen unfinished searches and two unreachable goals. A scene can hold an unannotated instance, or the agent can view an annotated one from outside its accepted viewpoints — one reason the ObjectNav tabs print SR at 1 m, 0.25 m and the official 0.1 m side by side. Visibility alone does not establish a successful approach.

L2

VLN failures are wrong targets, routes or stops. Of 44 VLN-CE failures, 15 pick the wrong target or route, 15 stop in the wrong place, seven leave the search unfinished and seven are blocked by obstacles. Some episodes revisit earlier locations and still miss the goal.

L3

EQA failures include wrong answers and disputed judgments. Of 19 rejected answers, six involve the wrong room or referent and three a visual misinterpretation; seven are answers the audit finds defensible from the recorded views or questions it finds ambiguous, and one has corrupted answer options.

fig · failure analysisopus-5.5 · one run per benchmark · ObjectNav at 1 m
Flow diagram of opus-5.5 failures. VLN-CE 44: wrong target or route, wrong stop, unfinished search, blocked. ObjectNav 63: 40 target visible, 5 wrong target, 16 unfinished search, unreachable. HM-EQA 19: 7 answer disputed, wrong answers, corrupted options.
◆Blue: agent errors · amber: evaluation or environment issues. Widths count episodes within each panel; categories come from model-assisted review.
L4

On a physical robot, reasoning transfers; body awareness does not. On a Unitree Go2 with the same monocular, primitive-action interface, across 31 hand-designed episodes the agent resolves logical and perceptual conditions before acting and completes some return routes (object pickup is assumed, not executed). Without body-state sensing it can turn too early and wedge the rear in a doorway; step() reports requested rather than realized motion, so under-rotation leaves a wrong heading; counting across views and revisits expose weak persistent spatial state.

go2 · ep 25 · successreal Unitree Go2 · fetch and deliver, person picked by an attribute
A real Unitree Go2 episode: 'Go to the remote control that fell on the floor behind you. Assuming you could pick it up, bring it to the person in the room wearing white shoes.' Four camera frames with the agent's reasoning: it turns around, spots the remote on the floor, pivots to the people, and stops in front of the seated person in white shoes. Faces are blurred.
◆Reasoning transfers. Two stages and an attribute: it turns 180° to find the remote, approaches it, then separates the seated person in white shoes from a standing person in gray shoes and delivers to the right one — 10 observe(), 10 step(), 52 primitives. Pickup is assumed, not executed or visually confirmed. Faces blurred.
go2 · ep 6 · failurereal Unitree Go2 · exit a door, then turn
A real Unitree Go2 episode: 'Exit through the door, turn left, and walk to the cardboard cutout of a person.' Four camera frames: it turns left while still in the doorway, the view fills with the wall edge, it believes it is facing back into the start room, and drifts into the left wall.
◆Body awareness does not. It turns the moment the camera clears the doorway, but the body behind the lens has not: the rear catches the wall edge, the agent misreads its heading as “looking back into the room”, and a blocked, disoriented spiral follows — never reaches the cutout, cut off after 38 primitives. The same start and instruction succeed in ep 7, where it clears the opening before turning.
L5

Read the numbers as stated. All scores are fixed 100-episode subsets and may differ on the full sets; each human reference is one tester. We cannot verify whether proprietary models were post-trained on these benchmarks — zero-shot describes how we use them, not their training history. Tool and harness comparisons mainly cover R2R-CE, several supplementary settings are single runs, and serving and context handling differ across harnesses.

06 — cite

Citation

If the probe, the board, or the agentic-embodied-control framing is useful in your research, please cite the paper.

@misc{zhou2026embodiedagentstakecontrol,
  title         = {Embodied Agents Take Control: Minimal-Interface Zero-Shot
                   Agents Rival Industrial-Scale Policies in
                   Vision-and-Language Navigation},
  author        = {Jian Zhou and Xunyi Zhao and Gengze Zhou and Zerui Li and
                   Sihao Lin and Jiajun Liu and Qi Wu},
  year          = {2026},
  eprint        = {2607.26148},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2607.26148}
}
Run agents on the same substrate → Read the paper AgentCanvas