ADR-observability-004

Date

2026-05-04

Status

accepted

Field

observability

Context

Errors raised during task execution disappeared into one of three holes: (a) per-node exceptions in graph_executor.py:337 were caught, written to log.exception, and stuffed into the node's output dict as {"error": str(e)} — never surfaced to the UI; (b) graph-level crashes at graph_executor.py:518 did broadcast nav_status {error: …} over WebSocket, but the frontend store.ts:267 handler read only status and dropped error; (c) every HTTP API failure (save, load, eval, plugin lifecycle, frontend React crashes) was handled with a local console.error that never reached the user. The frontend had no notification primitive — no toast, no panel, no badge — only a single red 10-px "error" word in the ExecutionToolbar. The user described this as "even when the backend hits a bug, the user has no idea". ExecutionLogger (ADR-observability-003) covers structured per-node I/O, not surfacing of failures.

Decision

Introduce one bus + envelope pipeline that every subsystem publishes to and that surfaces in four UI layers. (1) ErrorEnvelope dataclass (agentcanvas/backend/app/errors.py) — canonical schema with id, ts, severity, source, code, title, message, scope, details, hint. Mirrored verbatim in agentcanvas/frontend/src/errors.ts. (2) ErrorBus singleton — in-process pub/sub with a 200-entry ring buffer + a 500-deep async dispatch queue. publish() is thread-safe and loop-agnostic (uses loop.call_soon_threadsafe); subscribers are async, drained by a single dispatcher coroutine started in the FastAPI lifespan. (3) Producers — bus.from_exception(...) (auto-fills traceback) and bus.emit(...) (non-exception). Three call sites refactored: per-node executor catch, graph-level executor catch, FastAPI global Exception handler (returns 500 with envelope as JSON body so the frontend HTTP layer can pick it up). (4) LogBridgeHandler — logging.Handler subclass installed on the root logger at INFO+, converts every Python log record into an envelope with source="log". Skips agentcanvas.errors, uvicorn.*, httpx.*, httpcore.*, watchfiles.*, asyncio to prevent feedback loops and keep volume sane. Makes the frontend Report tab a true overall developer console without touching individual log sites. (5) error_event WS frame — bus subscriber at startup broadcasts every envelope as WSMessage(type="error_event", data=envelope.to_dict()) over the existing /ws channel. (6) REST backfill — GET /api/errors returns the ring buffer; frontend calls it on every WS (re)connect to recover events missed during disconnect. DELETE /api/errors clears, DELETE /api/errors/{id} dismisses one. (7) Frontend — Zustand useErrorStore (200-entry cap, dedup by id, unreadCount, focusReportTick signal); api.ts wraps fetch/post/put/delete so 4xx/5xx and network failures auto-ingest, recognising backend-supplied envelopes when present in the body; CanvasErrorBoundary.componentDidCatch reports React crashes into the same store. (8) UI surfaces — (i) Toast layer (components/ErrorToast.tsx) mounted at App root, max 3 visible, auto-dismiss 6 s for error / 4 s for warning, info/debug never toast; click → opens Report. (ii) ReportPanel as a fourth tab in the bottom OutputDrawer with severity filter chips, source pill, click-to-expand details/scope/traceback, pin/dismiss/clear, unread-count badge on the tab. (iii) ResizableBottomPanel auto-expands on focusReportTick. (iv) ReportBell in ExecutionToolbar with red unread badge; click triggers focusReportTick.

Alternatives

(a) Read d.error from nav_status and add a toast — the minimum patch. Rejected: only covers graph-level crashes; per-node failures, API errors, frontend crashes still invisible; "complete refactor" was the explicit ask. (b) Reuse exec_log (ADR-observability-003) — extend its frontend handler to surface entries with non-null error. Rejected: exec_log is per-node-firing structured I/O capture (suppressed under ExecutionPrinciples.suppress_ws in batch eval), not a general user-visible event channel; conflating the two would either flood the user with non-error firings or force a second filter layer; also doesn't cover API/frontend/log producers. (c) Sentry / OpenTelemetry — external error tracking. Rejected: research tool; adds external dependency; user wants events visible in the canvas UI during the run, not in a separate dashboard. (d) Sync subscribers + per-call await instead of a dispatcher queue — simpler. Rejected: the Python logging.Handler bridge fires from worker threads; loop.call_soon_threadsafe(queue.put_nowait, env) is the cleanest cross-thread bridge and keeps the publish hot path lock-free except for the ring-buffer append. (e) Persist envelopes to disk like ExecutionLogger does. Rejected for now: logging already covers audit trail in stderr; ring buffer is enough for in-session use; revisit if pinned events need to survive reload (deferred to Phase 3).

Rationale

One pipeline means one place for every producer to publish and one place for every UI surface to subscribe — no per-site error handling, no half-built notification systems competing. The envelope schema is the contract: severity drives whether to toast or just append; source drives the filter pills; scope.{node_id, step, execution_id} is what makes the eventual "jump to failed node" feature trivial in Phase 2; details.traceback is what makes the developer-console use case real. Bus dispatch is async over a queue so the publish path stays cheap (single ring-buffer append + one threadsafe call) — important because the logging.Handler bridge can fire from any thread (executor coroutines, server-mode subprocess proxies, batch eval workers). Re-using the existing /ws channel means zero new transport — the same connection, message envelope (WSMessage), and wsManager.on(type, handler) routing already used by nav_step / viewer_data / exec_log. The "Report" tab name keeps it distinct from the existing "Logs" tab (per-node execution trace structured by graph step), so the two never visually conflict.

Affected docs

glossary.md, real-time-observability.md, roadmap.md, decisions/observability/index.md, decisions/index.md