ADR-observability-005 — AgentCanvas

Date: 2026-05-10
Status: accepted ## Context DiscussNav's cost analysis surfaced that AgentCanvas had no end-to-end LLM token / USD accounting. First-pass remedy added a `usage_out: dict | None` parameter to `llm_complete` / `llm_complete_n` and required every LLM-using node to thread its own accumulator dict + `_self_log("usage_*")` calls. After implementing it for `LLMCallNode` and DiscussNav's three reasoning nodes (thought_fusion / decision_test / history_writer), the boilerplate cost was clear: every new nodeset author would re-implement the same 6-line accumulator pattern, VLM calls were forgotten entirely, and any code path that called `llm_complete` without remembering to plumb `usage_out=` silently lost data. User pushed back on the per-node plumbing as hardcode boilerplate and asked for a hook-based design that all graphs benefit from automatically. ## Decision LLM usage capture moves from per-node opt-in plumbing to an executor-owned cross-cutting hook keyed by Python's `contextvars`. 1. **`app.llm.call._current_node_usage`** — module-level `ContextVar[dict | None]` with `default=None`. 2. **`_accumulate_usage(response)`** — called inside `llm_complete` / `llm_complete_n` / `vlm_complete` after every successful `litellm.acompletion`. Reads the bucket from the contextvar; if `None` (call originated outside a graph node — CLI, key validator, etc.) the helper no-ops. Otherwise, accumulates `{calls, model, prompt_tokens, completion_tokens, total_tokens, cached_tokens, usd_cost}` into the bucket. Cost comes from `litellm.completion_cost(response)` so the framework never owns a model→price table. 3. **`GraphExecutor` lifecycle wrap.** Around each `instance.forward(fire_inputs, ctx)` call: ```python usage_bucket: dict = {"calls": 0, "model": "", ...} usage_token = _current_node_usage.set(usage_bucket) try: result = await instance.forward(fire_inputs, ctx) finally: _current_node_usage.reset(usage_token) if usage_bucket["calls"] > 0: usage_bucket["usd_cost"] = round(usage_bucket["usd_cost"], 6) instance._log_buffer.append({"key": "usage", "value": usage_bucket}) ``` 4. **Result shape.** Every node firing that touched the LLM gets exactly one `{"key":"usage","value":{...}}` entry in its inner log, picked up by `instance.log()` and written into the run log alongside other `_self_log` entries. Zero entries when `calls == 0`. 5. **Nodesets touch zero usage code.** `LLMCallNode`, DiscussNav, MapGPT, NavGPT, future reasoners — none of them know the hook exists. Authors call `await llm_complete(...)` and the framework handles aggregation. ## Alternatives a. **Per-node `usage_out: dict` plumbing.** This is what shipped first. Rejected after-the-fact: 6 lines of boilerplate per call site, easy to forget, missed VLM, hostile to third-party nodesets. b. **litellm `success_callback` / `CustomLogger`.** litellm offers a built-in callback that fires on every `acompletion`. Considered for max coverage (would intercept any litellm call, even from a third-party library imported into a node). Rejected for v1 because (i) we don't currently have third-party libraries calling litellm inside our process, (ii) the CustomLogger API has more surface area to maintain than a contextvar, and (iii) the contextvar approach is one-line-portable to litellm callbacks if we ever need that breadth — keep the executor wrap, just have the callback be the writer. c. **Background telemetry (Langfuse / W&B / OpenTelemetry).** Out of scope for this ADR. The contextvar bucket is a minimal "data exists in the run log" foundation; an upstream telemetry backend can subscribe to `_log_buffer` events later without disturbing this hook. ## Rationale - **Cross-cutting concerns belong at the executor, not in every node.** Latency budgets, model-swap audit, span tracing all want the same lifecycle handle (`enter execute → exit execute`). Putting one ContextVar set/reset around `instance.forward(...)` makes the executor the single point where the framework owns the lifecycle, and the LLM helpers are pure consumers. - **`contextvars` propagates correctly across `await`.** Python's contextvars are designed for exactly this: the value set in the executor before `await instance.forward(...)` is visible inside any `await llm_complete(...)` reached during that execute, even across multiple `await` boundaries — because each Task inherits the calling context. No threading, no globals, no manual context plumbing. - **`litellm.completion_cost` makes USD a free field.** litellm carries an internal model→price table updated with each release, so we don't have to maintain pricing for 23 providers. `try/except` around the cost call ensures unknown models (local vLLM, custom endpoints) report `usd_cost: 0.0` rather than crashing the hook. - **Asymmetric default (`None` outside graph nodes) prevents accounting pollution.** CLI calls, key-validator probes, and direct `python -c "...llm_complete(...)..."` smoke tests don't accidentally accumulate into a stale bucket, because `_accumulate_usage` checks `is None` and bails out. ## Affected docs - `developer-guide/capabilities/observability.md` — LLM usage entry shape added - `developer-guide/core/glossary.md` — "Usage hook" term added - `developer-guide/design-docs/llm-config.md` (if present) — usage section
Field: observability