ADR-eval-004
Per-episode log storage and replay-parser ABC simplification
Context
Batch eval persisted node-firing logs and image assets at run-level granularity: outputs/eval_runs/{run_id}/log.jsonl + outputs/eval_runs/{run_id}/assets/, both flat files / dirs shared across every episode and every worker. The mechanism was one ExecutionLogger per BatchEvalRunner, created once in __init__ (agent_loop/eval_batch.py) and threaded through to every per-episode LoopRunner instance. Per-episode identity did exist — LoopRunner._execution_id = f"{run_id}_ep{idx}_w{worker_id}" was set at eval_batch.py:522 for WS broadcast tagging — but it was never threaded into ExecutionLogger.log_node(). Every persisted NodeLogEntry was stamped execution_id = self._logger.execution_id = run_id, the same value across every episode × worker. With worker_count > 1 the per-firing append order in log.jsonl interleaved by wall-clock, and the step field reset to 0 inside each episode but those resets overlapped between workers, so neither the execution_id field nor step could recover episode membership. Assets were flat under one assets/ dir with names like s{step}_f{firing_idx}_{label}__{port}.jpg where firing_idx was the logger-global monotonic counter — unique on disk but with no episode prefix, so a 100-episode 3-worker run produced one bucket of ~thousands of files with no grouping. The downstream consequence: per-episode reconstruction from outputs/eval_runs/{run_id}/ was effectively impossible. The replay subsystem made this concrete — app/replay/interface.py:GenericReplayParser._split_episodes walked the flat log.jsonl and cut at every __reset node firing to recover episode boundaries, and its own docstring (interface.py:220-233) admitted the failure mode: "on multi-worker eval runs the log entries interleave, and _split_episodes cuts at every __reset. The first few episodes can collapse to 1 entry each while one downstream episode absorbs all the mixed step entries. That's a logger-side limitation (no worker_id tag)". The single-execution canvas path did not have this problem because agent_loop/loop_runner.py:117-132 self-creates a per-execution ExecutionLogger(persist_dir=outputs/runs/{execution_id}/) when none is injected — every canvas run is self-contained. The eval batch path was the architectural regression: it injected a shared logger to amortize one ring buffer across many episodes, and that amortization broke per-episode addressability.
Decision
Make ExecutionLogger lifecycle per-episode in batch eval, mirroring what canvas single-run already does. The flat log.jsonl + assets/ directly under the run dir cease to exist; each episode owns a self-contained subdir. Six rules.
(1) Episode dir as the unit of log storage. Layout under outputs/eval_runs/{run_id}/:
spec.json shared_urls.json summary.json stdout.log stderr.log _DONE
graph.json # run-level snapshot
episodes/
ep{idx:04d}/
log.jsonl # this episode's firings only
assets/ # this episode's images only
episode.json # self-describing row from summary.json
graph.json, summary.json, spec.json, shared_urls.json, stdout.log, stderr.log, _DONE stay at run level (one graph and one job spec for the whole run). Everything firing-level moves into the episode subdir. The four-digit zero-padded index (ep0000, ep0001, …) makes lexicographic sort match numeric sort up to 10000 episodes.
(2) Per-episode ExecutionLogger constructed in _run_one_episode (agent_loop/eval_batch.py:_run_one_episode). The episode dir is created up front so even an early-failure episode (env on_load throws before the LoopRunner is built) gets an episode.json written at episode end. The logger's execution_id is the composite {run_id}_ep{idx:04d} — every NodeLogEntry in log.jsonl naturally carries this id without log_node() taking a new parameter. Asset filenames keep their existing shape (s{step}_f{firing}_{label}__{port}.jpg); the per-episode firing counter starts from 0 and is bounded by step_budget, so collisions are not a concern within an episode dir.
(3) BatchEvalRunner owns run-level artifacts only. __init__ writes graph.json once; the deleted shared logger is not replaced. The per-LoopRunner graph.json write at loop_runner.py:139-146 is gated to self-created (canvas) loggers — eval's injected logger sits in an episode dir, and dropping a run-level snapshot into every episode dir would be N-way redundancy. Canvas path is unchanged; eval path's graph.json now lives exactly once.
(4) episode.json per episode dir, written from eval_storage.episode_to_dict(episode). Schema: episode_index, episode_id, scene_id, instruction, metrics, step_count, elapsed_sec, status, error, selectors, worker_id. run_to_dict in eval_storage.py is refactored to call episode_to_dict so the per-episode file and the embedded episodes[] list in summary.json share one source of truth — changing the episode shape touches one function. A single episode dir is consumable by replay / debugging / architect tooling without joining back to run-level summary.json.
(5) run_id is a second-precision timestamp with filesystem-collision suffix. JobScheduler._fresh_run_id() returns time.strftime("%Y%m%d_%H%M%S") (e.g. 20260515_121636); if the dir already exists, append _2, _3, … until unique. Replaces the previous uuid.uuid4().hex[:8]. Two motivations: (a) human-readable ordering — ls outputs/eval_runs/ is chronological by name, no stat needed; (b) cheap external correlation — finding which run was running at a given wall-clock moment is now a string-prefix comparison.
(6) BaseReplayParser.parse ABC simplified to take a single-episode log path. New signature: parse(self, episode_log_path: Path) -> ReplayEpisode. The episode_index parameter is gone — the file is the episode. Five methods are deleted from the ABC and GenericReplayParser: _split_episodes, _iter_episode_boundaries, episode_count, episode_summaries, is_episode_start. _pick_frame, _collect_info, _is_short_value (the per-step assembly logic) are unchanged. get_smooth_frame signature loses the episode_index parameter for the same reason. replay/router.py:list_episodes reads summary.json directly for the picker (no parser re-derivation, no possibility of disagreement). Replay payload gains an execution_id: "{run_id}_ep{idx:04d}" field so the frontend can build /api/logs/{execution_id}/assets/{filename} URLs through the existing logs endpoint — no new asset route, one addressing scheme for canvas and eval. Plugin parsers (one in tree: workspace/nodesets/server/hmeqa_replay.py) update to the new signature; out-of-tree parsers would break, but the workspace shipped exactly one.
Out of scope (deferred): (i) Migration of pre-2026-05-15 eval runs — old flat-layout runs remain on disk but are invisible to /api/logs listing and /api/replay (both scan episodes/). The user explicitly accepted this trade in exchange for not building a migration script. (ii) Collapsing the outputs/eval_runs/ ↔ outputs/design_runs/{graph}/vN/iter_M/ redundancy where architect iters copy episodes/summary into iter dirs — a separate contract refactor. (iii) .exp.yaml's step_budget field unifying with the graph-internal step_budget (the drift surfaced again during the verification run for this ADR) — also separate. (iv) Batch-level log.jsonl — not preserved; a glob over episodes/*/log.jsonl reconstructs the "everything" view on demand, and nothing in the codebase consumes a true batch-merged stream.
Alternatives
(a) Thread execution_id into log_node() and keep one shared logger writing to one flat log.jsonl — rejected: leaves the logger as a multi-writer multiplexer (N write queues keyed by execution_id), leaves assets/ flat (still no episode grouping unless filenames get an episode prefix too), and doesn't recover the architectural symmetry with the canvas single-run path that already did per-execution dirs correctly. The simpler refactor is to remove the regression, not paper over it with a router. (b) Per-episode log files but keep a run-level merged log.jsonl as an additional artifact — rejected: no consumer needs the merged view, and writing it requires either a second logger or a post-run merge pass; both add code and disk for negative value. (c) Keep the flat layout and add episode_index + worker_id fields to NodeLogEntry so consumers demux by filter — rejected: keeps _split_episodes machinery alive in the parser (which still has to scan the whole file to find the right episode's rows), still leaves assets ungrouped on disk, and tools that operate on episode dirs (architect's "copy iter_M's logs") still have nothing self-contained to copy. (d) Canvas-style execution_id-based path (outputs/eval_runs/{execution_id}/{log.jsonl,assets,graph.json} with one dir per episode at top level, no nesting) — rejected: loses the run-level grouping ("which episodes were in run X") that downstream tooling depends on; summary.json would need to point at sibling dirs by name, and the addressing scheme for run-level artifacts (spec.json, summary.json) breaks. The run/{episode} nesting is the natural fit. (e) UUID-with-timestamp-prefix run_id (e.g. 20260515_121636_abcd1234) — rejected: longer name with no information payoff once filesystem-collision suffix handles same-second submits. (f) Migrate old runs — rejected: the user opted out explicitly; old runs are historical artifacts, not active reference data, and a migration script would be ~50 lines of code for one-time use.
Rationale
The root cause was lifecycle mismatch — ExecutionLogger was constructed at batch scope but the meaningful unit of reconstruction is the episode. Two consumers needed per-episode addressability: replay (cf. its own self-documented breakage) and the user's debugging workflow (the question that prompted this ADR). The decision realigns logger lifecycle with the consumption unit, and once that alignment is made, three things fall out for free: (i) the replay parser stops needing boundary-detection code (the file IS the episode, ~80 lines of _split_episodes machinery deletes); (ii) the canvas and eval log paths converge on one mental model — "one execution = one logger = one dir"; (iii) assets/ collisions stop being a theoretical concern. The timestamp run_id (rule 5) is a small independent choice bundled here because the layout-touching ADR was the natural moment to make ls outputs/eval_runs/ chronological — searching by find -newer-style probes was the friction it removes. The replay ABC break (rule 6) is a deliberate breaking change scoped to one workspace plugin; the trade-off was "preserve plugin contract but leave dead parameters" vs "clean signature, update one file" — the latter is the actual definition of "clean". Backward compatibility is the noisiest rejected alternative (f): the user's choice to skip migration is the correct one for a research codebase where eval runs are inputs to a specific paper question, not historical data that future analyses depend on; preserving them on disk under their original layout is sufficient for forensic recovery while not constraining the new code path.
Affected docs
developer-guide/core/architecture.md— eval log layout sectiondeveloper-guide/core/codebase-map.md— new file paths underoutputs/eval_runs/, no source-file renamesdeveloper-guide/core/glossary.md— new terms (episode dir, episode.json, per-episode logger)developer-guide/core/roadmap.md— mark "per-episode log reconstruction" as Donedeveloper-guide/core/decisions/eval/index.md— table rowdeveloper-guide/core/decisions/index.md— count bumpdeveloper-guide/capabilities/real-time-observability.md— log layout narrativedeveloper-guide/design-docs/batch-eval.md— eval output layoutdeveloper-guide/design-docs/execution-logs.md— log file location, replay parser shapedeveloper-guide/tutorials/execution-logs.md— tutorial pathsdeveloper-guide/tutorials/batch-eval.md— tutorial output references.claude/PROJECT_OVERVIEW.md— flag for next/overview:update
Follow-up: A future ADR may collapse the outputs/eval_runs/{run_id}/episodes/ep{NNNN}/ ↔ outputs/design_runs/{graph}/vN/iter_M/episodes/ two-tree redundancy by switching architect skills from "copy into iter dir" to "pointer + read-through". A second future ADR may unify .exp.yaml's step_budget with the graph-internal step_budget so /experiment:run and /architect:* agree on the cap. Neither is blocking — both surfaced during this work but are about different subsystems.