ADR-observability-003
Node output log system — two-layer execution logging
Context
Both canvas graph execution and batch eval produce per-node events (firings, errors, state changes) that were only logged to stdout via Python logging.info() — not persisted, not structured, not queryable after process exit. Debugging agent reasoning required re-running graphs. The user specifically needed visibility into LLM prompts assembled inside node.forward(), which the executor cannot see from outside.
Decision
(1) Two-layer capture: automatic exterior (executor wraps every _fire_node() with timing + port I/O capture) and voluntary interior (nodes call self._self_log(key, value) inside forward() for domain-specific details). (2) Add _log_buffer, _self_log(), and log() to BaseCanvasNode — log() is overridable for filtering/transforming. (3) ExecutionLogger with collections.deque ring buffer (2000 entries), JSONL file persistence to outputs/runs/ (canvas) or outputs/eval_runs/ (eval), and log_serialize() that recursively truncates large strings >1KB and summarizes numpy arrays/tensors. (4) exec_log WS event broadcasts summaries per node firing (suppressed in eval mode via suppress_nav_events). (5) REST API at /api/logs with 4 GET endpoints for querying. (6) Frontend LogPanel in OutputDrawer with timeline view grouped by step, expandable I/O, filterable. (7) Logger injected via LoopRunner(logger=) → GraphExecutor(logger=) chain; canvas auto-creates logger; eval creates one per EvalRun.
Alternatives
(a) Hook-based logging — register a built-in "logging hook" that captures I/O via the existing PreNodeExecute/PostNodeExecute hook system. Rejected: hooks are for user-configurable behavior, not framework infrastructure; timing would be less precise (hooks fire after execute, can't wrap it); double-serialization overhead. (b) Automatic-only port I/O capture without _self_log() — rejected: cannot see node-internal details like assembled LLM prompts, API parameters, or token counts. (c) OpenTelemetry tracing — rejected: over-engineered for a research tool; adds external dependency for no benefit.
Rationale
_fire_node() is the single chokepoint where all nodes execute — hooking there gives 100% automatic coverage with ~10 lines. The voluntary _self_log() layer lets node authors expose domain-specific internals (prompts, rewards, token counts) without the executor needing to understand node semantics. JSONL is simpler than SQLite for append-heavy, read-rarely logs — no schema migrations, no connection management, and directly grep/jq-able from the terminal. The exec_log WS event is suppressed in eval mode to prevent flooding (500K+ messages for large batch runs).
Affected docs
glossary.md, architecture.md, roadmap.md