Embodied agents
take control

Who holds the control loop?

Embodied navigation has answered this question two ways. Trained policies bake the loop into weights — capable, but offering limited flexibility across environments and weak recovery when execution goes wrong. Workflow systems keep the loop in human-authored code: the foundation model is one reasoning module inside a hand-wired pipeline of maps, memory, planners, and verifiers — the dominant zero-shot recipe — and even recent dual-brain systems fix the handoff between reasoning and control.

We study the case where the model itself holds the loop — agentic embodied control: the reasoning model directly steers every action through plain tool calls, so reasoning and control stay aligned with no fixed handoff. It is the organization coding agents already use to write software; frontier models' multimodal, multi-turn, and spatial abilities may have made it viable for embodiment. The probe asks how far it goes with no navigation machinery at all.

fig 1who holds the loop · and how far that gets it

One camera. Basic actions. Nothing else.

The minimal interface is a diagnostic lower bound, not a convenience baseline. Everything a navigation system usually leans on is deliberately taken away; whatever performance remains must come from the model itself. If the agent succeeds under these conditions, no hidden navigation module can claim the credit — and where it fails, the missing capability is named.

Three coding-agent harnesses drive it — Claude Agent SDK, Codex CLI, and the small open-source mini-swe-agent — each connected to the simulator through the same two tools. The entire task specification is one system prompt:

You are controlling a robot in a real indoor environment (a photorealistic
3D scan of a building). You interact only through these tools:

- observe(): look through the robot's forward-facing camera (returns an RGB image).
- step(actions): execute movement actions in order. 0 = STOP (permanently ends
  the episode — declares you have reached the goal), 1 = move forward 0.25 m,
  2 = turn left 15 degrees, 3 = turn right 15 degrees.

Your task is to follow this navigation instruction to its endpoint:

"{instruction}"

Rules:
- Alternate observing and stepping: look, decide where the instruction wants
  you to go next, move, look again.
- You have a budget of 500 movement actions.
- You succeed only if you issue action 0 (STOP) while within 3 meters of the
  instruction's endpoint. STOP is permanent — issue it only when you believe
  you are at the goal.
- Turning in place (e.g. step([2,2,2,2,2,2])) is a cheap way to look around
  when unsure.
- Work autonomously until you stop; nobody can answer questions.

A real episode, decision by decision

This is episode 10 of the board, replayed from its log — every frame is the exact image the agent saw, every quote is its own recorded reasoning, every step() is the exact call it issued. It turns around, fights its way past the bed corner, wanders into the wrong room, rebuilds its mental map out loud, and stops — correctly — in the bathroom. The paper's Fig. 2 shows this same episode.

replay — mini-swe-agent × fable-5 · ep 10 · R2R-CE val-unseen// from episode_10.jsonl

instruction “Turn around and walk to the end of the bed. At the end of the bed turn right and walk to the hallway. Once at the entrance way, walk through the hallway to the next room.” SUCCESS · stops 2.98 m from goal

obs 1 / 20

▶ play

They add machinery. We expose what the model already knows.

On R2R-CE, engineered zero-shot agents top out at 55 SR with panoramas, depth, maps, memory, and action tools. The best industrial-scale trained policy reports 72 with panoramic input and in-domain data. The minimal-interface agent — one camera, four actions, no training — reaches 72 with a small open-source harness and 78 at maximum effort.

Model, harness, interface

The board factorizes the system into three layers and moves one at a time. The answer: model choice dominates, harness differences are modest and descriptive, and the interface is the capability boundary — decisive compensation for weaker models, optional acceleration for the strongest. Model-centered, but not model-only.

capability.axesmove one layer at a time

Where it still fails

The probe is a diagnostic; its failures are as informative as its wins.

Citation

If the probe, the board, or the agentic-embodied-control framing is useful in your research, please cite the paper.

@misc{zhou2026embodiedagentstakecontrol,
  title         = {Embodied Agents Take Control: Minimal-Interface Zero-Shot
                   Agents Rival Industrial-Scale Policies in
                   Vision-and-Language Navigation},
  author        = {Jian Zhou and Xunyi Zhao and Gengze Zhou and Zerui Li and
                   Sihao Lin and Jiajun Liu and Qi Wu},
  year          = {2026},
  eprint        = {2607.26148},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2607.26148}
}