ADR-eval-001
Eval page redesign — graph-driven batch evaluation via LoopRunner
Context
The eval page had its own execution pipeline (eval_orchestrator.py with VLNCERunner in a background thread), separate from the canvas execution system. It only supported neural policy checkpoints, had no graph support, no output viewers, no run history, and used os.chdir() to resolve VLN-CE paths. Any improvement to the canvas execution engine (state containers, hooks, parallel nodes) had to be separately ported to eval.
Decision
(1) Replace the standalone eval pipeline with a BatchEvalOrchestrator that wraps LoopRunner + GraphExecutor — the same engine the canvas uses. Each episode gets a fresh LoopRunner instance for clean state. (2) Add ExecutionPrinciples dataclass to configure batch mode (no_pause, suppress_nav_events, source_tag) without changing LoopRunner's default canvas behavior. (3) Add ExecutionGuard with ExecutionMode enum — exclusive lock preventing canvas and eval from running simultaneously (HTTP 409). (4) Add env metadata interface (BaseNodeSet.get_eval_metadata()) so env nodesets expose splits, episode counts, and metrics. (5) Add registry-backed graph introspection (detect_env_nodesets_for_graph()) with prefix fallback for nodesets that fail to import. (6) Consolidate nodeset auto-loading into WorkspaceComponentRegistry.ensure_nodesets_for_graph() — shared by both canvas and eval. (7) New eval API at /api/eval/v2 with run persistence to outputs/eval_runs/. (8) New frontend eval page with graph selector, env introspection, live progress, sortable episodes table, metric cards, and run history.
Alternatives
(a) Eval mode flag inside GraphExecutor — rejected: violates SRP, risks destabilizing canvas, entangles episode/iteration boundaries. (b) Enhance existing eval_orchestrator.py with graph support — rejected: still maintains separate execution engine, no viewer reuse, no canvas improvement sharing. (c) Subprocess-based eval isolation — rejected: adds IPC complexity, the ExclusionGuard is simpler for a single-user research tool.
Rationale
Batch eval is fundamentally "run a graph N times for N episodes." The LoopRunner already knows how to run graphs. Wrapping it in an episode loop (BatchEvalOrchestrator) reuses all existing infrastructure — graph execution, state containers, hooks, node handlers — without duplication. ExecutionPrinciples keeps the coupling minimal (4 fields, all optional). ExecutionGuard prevents the only real concurrency hazard (shared env singleton). Fresh LoopRunner per episode guarantees clean state.
Affected docs
roadmap.md, architecture.md, glossary.md