2026-07-12

Rollout Loop vs. Agent Loop: Naming the Two Clocks of an Embodied Agent

This post does one thing: pin down the loop vocabulary for embodied agents. "Loop" crossed from the LLM-agent world into embodied AI and quietly changed meaning on the way — a coding agent's loop ticks on LLM turns, an embodied agent's loop ticks on environment steps, and calling both "the agent loop" hides the difference. Two terms and one composition rule: the rollout loop runs on the environment's clock, the agent loop (ReAct [1]) runs on the reasoner's clock, and a real system couples the two — one nested inside the other, or both running side by side.

1Two Loops, Two Clocks

rollout loop agent loop observe body act env.step one iteration = one environment step ends when the episode ends body: frozen VLA · scripted controller · LLM — anything reason call tool read result one iteration = one LLM turn ends when the task is judged done no environment — tool results are the world (ReAct) shaded = who owns the clock: the environment on the left, the reasoner on the right

The rollout loop is not a design choice — embodiment imposes it. Whatever sits in the body is stepped by the environment, once per tick, until the episode ends; a frozen policy like OpenVLA [2] or π0 [3] loops thousands of times with zero reasoning anywhere in the cycle. The agent loop is the opposite in every column: ReAct [1] made it canonical, the LLM turn is the iteration, and no environment is required — coding agents run it for hundreds of turns with no world attached.

2They Compose: Nested or Concurrent

Four couplings recur, in two families. Nested — one loop contains the other and blocks on it: the agent loop as the rollout's policy (1), or the rollout as the agent's tool (2). Concurrent — neither waits: coupled by buffers (3), or gated by events (4).

rollout loop — the environment's clock observe agent loop — the reasoner's clock 0…N turns per env step reason call tool read act env.step next tick — until the episode ends 0 turns · frozen policy — a pure rollout (a VLA eval) 1 turn · one LLM call per step — the NavGPT shape N turns · a discussion per step — the DiscussNav shape

1 · Agent-as-policy: the agent loop sits in the rollout's policy slot, and env.step blocks while it deliberates — the natural shape wherever the world can hold still (simulation, turn-based settings). One number governs it: LLM turns per environment step — zero, one turn every K steps, or many per step. (A turn may emit one action or a K-step chunk; as long as the world waits for the turn, it is still this coupling.)

notes — who sits where on the 0…N dial

NavGPT [4] fixes it at one; DiscussNav [5] spends a multi-expert discussion before every step; a VLA eval spends zero. The corners are the degenerate cases — rollout-only (a frozen policy), agent-only (a coding agent, no world) — and everything between is legitimate design space. In AgentCanvas [6] the distinction is structural: the iteration primitive is deliberately called just a loop, whether it is a rollout loop or an agent loop is decided by what you draw in its body, and nesting is scope-in-scope — the ordinary case, not a special one.

agent loop — the reasoner's clock reason rollout — one tool call observe act env.step read result the whole episode ticks once on the agent's clock — SayCan at skill granularity · Reflexion at episode granularity

2 · Rollout-as-tool — the mirror of 1: the environment's clock sits inside the reasoner's, and the rollout serves as a tool of the agent loop exactly as the agent loop in 1 serves as the rollout's policy. One whole rollout per tool call — at skill granularity (SayCan [7]: each "execute skill" is a closed-loop rollout) or at episode granularity (Reflexion [8]: reason, launch a rollout, read the log, go again).

notes — works with this shape

Voyager [9] grows a skill library by launching Minecraft rollouts and reading what happened; Eureka [10] has GPT-4 write reward code, launch full RL training runs as tool calls, and iterate on the rollout statistics. At the largest granularity this is the shape of any automated design or policy search in which an LLM operates the experiment loop.

The concurrent family next: nothing waits.

rollout loop — fast, free-running agent loop — slow, free-running observe act env.step ticks every control step — does not wait reason call tool read ticks every LLM turn — seconds or more latest plan latest obs neither loop waits for the other — the coupling is a buffer, not a call

3 · Parallel: real time does not wait for a turn. Both loops free-run on their own clocks; the coupling is a pair of buffers — act on the latest plan, reason over the latest observation. The exchange rate is no longer designed: it is the ratio of the two clock speeds.

notes — the same shape in robots and in speech

Dual-system robots are this coupling: a fast System-1 controller with a slow System-2 reasoner riding alongside (Figure's Helix, NVIDIA's GR00T N1), the controller at tens of hertz over a reasoner at seconds per turn. Speech hit the same fork and named it first: turn-based dialogue is the nested coupling, and full-duplex models — listening while speaking, as in Kyutai's Moshi — are the parallel one.

boundary case — pipelined execution (1's schedule, 3's concurrency)

A policy needs ~100 ms per inference and emits a one-second action chunk (chunking: ACT [11]; π0 [3] acts this way). Run blocking, this is coupling 1: move a second, freeze, move again. Pipelined execution overlaps the two — compute plan k+1 while chunk k runs — so motion never stops, and the plan in use is always exactly one chunk stale. Real-Time Chunking [12] is this, made to work for VLAs: freeze the actions certain to execute, inpaint the rest so the boundary stays smooth; real-time MPC has run this way for decades. The exchange pattern is still 1's, only the concurrency is 3's — a boundary case, not a mode.

act think chunk k chunk k+1 chunk k+2 … plan k+1 plan k+2 plan k+3 … time act ticks on the env clock, think on the LLM clock — the plan in use is always exactly one chunk old
rollout — fast policy, free-running observe act env.step agent loop — dormant (wakes on event) stuck / uncertain / human new plan between events the loops do not exchange at all — a safe fallback holds the world

4 · Event-triggered: concurrency on a condition. Three pieces: the rollout free-runs on a fast policy; the agent loop is not running; a monitor watches for a defined trigger — failure, uncertainty crossing a threshold, a human interjecting. On the trigger the agent loop wakes, runs as many turns as it needs, and hands back a new plan while the rollout keeps itself safe (hold, or follow the old plan). Unlike 3 there is no continuous flow — and no period: between events the coupling is exactly zero.

notes — works with this shape

Inner Monologue [13] replans when success detection or human feedback flags a problem — feedback events gate the LLM's involvement. KnowNo [14] makes the trigger principled: conformal uncertainty decides when the planner must stop and ask a human for help. The escalation pattern — run the fast policy until something goes wrong, then wake the reasoner — is the deployed shape of most "VLA with an LLM fallback" stacks.

Mode Exchange initiated by Who waits Examples
1 · Agent-as-policy The rollout, every step. The world. DiscussNav in simulation.
2 · Rollout-as-tool The agent, per skill or per episode. The agent. SayCan · Reflexion.
3 · Parallel No one — buffers couple the loops. Neither. Helix · GR00T N1 · full-duplex dialogue.
4 · Event-triggered An event raised by the rollout. Neither — a fallback holds. A VLA that escalates to an LLM when stuck.

3The Definitions

Rollout loop Agent loop
One iteration One environment step. One LLM turn.
Terminates when The episode ends — success, failure, or step budget. The task is judged done, by the model or its harness.
Loop body Anything mapping observation → action. Reason → tool call → read result.
Needs an LLM? No — the body can be a pure function. Yes — the LLM turn is the iteration.
Needs an environment? Yes — the env's transition is the loop. No — tool results are the world.
Budget currency Env steps. LLM turns / tokens.

4Position

Reserve agent loop for the LLM-turn loop and call the embodied interaction loop the rollout loop. An embodied agent always has the latter and only sometimes the former. The interesting design space is the coupling between the two clocks — how much deliberation each environment step buys, and whether the world waits for it (the nested couplings) or does not (the concurrent ones). A system description should say which clock its loop runs on, and which coupling it assumes.
References ([1]–[14])

[1] S. Yao et al., "ReAct: Synergizing reasoning and acting in language models," in Proc. Int. Conf. Learn. Representations (ICLR), 2023. arXiv:2210.03629.

[2] M. J. Kim et al., "OpenVLA: An open-source vision-language-action model," in Proc. Conf. Robot Learn. (CoRL), 2024. arXiv:2406.09246.

[3] K. Black et al., "π0: A vision-language-action flow model for general robot control," Physical Intelligence, 2024. arXiv:2410.24164.

[4] G. Zhou, Y. Hong, and Q. Wu, "NavGPT: Explicit reasoning in vision-and-language navigation with large language models," in Proc. AAAI Conf. Artif. Intell. (AAAI), 2024. arXiv:2305.16986.

[5] Y. Long et al., "Discuss before moving: Visual language navigation via multi-expert discussions," in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024. arXiv:2309.11382.

[6] J. Zhou, "AgentCanvas: A typed-graph runtime for embodied agents," 2026. [Online]. Available: docs — Graph-Expressible Agents.

[7] M. Ahn et al., "Do as I can, not as I say: Grounding language in robotic affordances," in Proc. Conf. Robot Learn. (CoRL), 2022. arXiv:2204.01691.

[8] N. Shinn et al., "Reflexion: Language agents with verbal reinforcement learning," in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2023. arXiv:2303.11366.

[9] G. Wang et al., "Voyager: An open-ended embodied agent with large language models," arXiv:2305.16291, 2023.

[10] Y. J. Ma et al., "Eureka: Human-level reward design via coding large language models," in Proc. Int. Conf. Learn. Representations (ICLR), 2024. arXiv:2310.12931.

[11] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, "Learning fine-grained bimanual manipulation with low-cost hardware," in Proc. Robot.: Sci. Syst. (RSS), 2023. arXiv:2304.13705.

[12] K. Black, M. Y. Galliker, and S. Levine, "Real-time execution of action chunking flow policies," in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2025. arXiv:2506.07339.

[13] W. Huang et al., "Inner Monologue: Embodied reasoning through planning with language models," in Proc. Conf. Robot Learn. (CoRL), 2022. arXiv:2207.05608.

[14] A. Z. Ren et al., "Robots that ask for help: Uncertainty alignment for large language model planners," in Proc. Conf. Robot Learn. (CoRL), 2023. arXiv:2307.01928.

← Back to blog