Execution Logs
Two-layer logging โ executor exterior + node interior, JSONL, REST API
The execution log system (ADR-observability-003) captures per-node I/O for every firing during graph execution. It is two layers: the executor automatically records port data and timing (the exterior layer โ zero node-author effort), while a node may voluntarily emit domain-specific details like LLM prompts and token counts (the interior layer, opt-in). Logs persist as one JSONL line per firing with images saved as sidecar assets โ queryable over REST and replayable in the frontend Log Viewer. This page owns the record shape, serialization, and storage layout; the live viewer is covered under Execution Logs & Live Views.
"Two layers" is the mental model. The first figure draws it; the second shows how the code actually realizes it (they differ โ see the note).
Mental model โ two layers, one entry.
How the code does it โ one log_node call, two data sources in, several sinks out.
logger.py ยท ExecutionLogger.log_node, called once per firing from graph_executor.py ยท _fire_node โ that receives the exterior data (ports, timing, result) and the interior data (instance.log(): _self_log entries plus the merged per-node usage bucket) together. There is no separate executor-level log call. From that one entry the data fans to several sinks: an in-memory ring buffer (REST queries), a write queue flushed to log.jsonl (durable), image/depth ports saved under assets/, and a separate per-firing exec_log WebSocket broadcast (live; suppressed in eval mode).
1. What the log captures โ two layers
Every node firing produces a NodeLogEntry with data from both layers:
| Layer | Who writes | What's captured | Example |
|---|---|---|---|
| Automatic (exterior) | GraphExecutor after _fire_node() |
Port inputs/outputs, timing, wire types, errors | {inputs: {rgb: <asset>}, outputs: {action: 1}, duration_ms: 25} |
| Voluntary (interior) | Node calls self._self_log(key, value) inside forward() |
Domain-specific details invisible to the executor | LLM prompt text, API response, token count, format hints |
The exterior layer requires zero effort from node authors โ the executor instruments every firing automatically. The interior layer is opt-in: nodes that want to expose internal reasoning call _self_log() during execution.
1.1 The smallest thing a node logs
BaseCanvasNode provides two methods in app/components/bases.py:
def _self_log(self, key: str, value: Any) -> None: """Record an internal detail during forward(). Call inside forward() to log domain-specific data that the executor cannot see from the outside (e.g. assembled LLM prompts, API responses). """ self._log_buffer.append({"key": key, "value": value}) def log(self) -> list[dict]: """Return voluntary log entries from the last forward() call. Called by the executor after forward() completes. Override to filter, transform, or add computed summaries. """ return self._log_buffer
Example usage in a node's forward():
async def forward(self, inputs: dict, ctx: Any) -> dict: prompt = self._build_prompt(inputs["instruction"], inputs["history"]) self._self_log("prompt", prompt) # voluntary: log assembled prompt response = await call_llm(prompt) self._self_log("response", response.text) # voluntary: log raw response self._self_log("tokens", response.usage) # voluntary: log token counts return {"action": parse_action(response)} # executor automatically logs inputs, outputs, and timing
2. Backend: ExecutionLogger
ExecutionLogger (app/logging/logger.py) manages in-memory buffering and JSONL persistence.
2.1 Data Model
Two Pydantic models in app/logging/models.py:
NodeLogEntry โ one node firing:
| Field | Type | Description |
|---|---|---|
timestamp |
datetime |
UTC time of firing |
execution_id |
str |
Execution run identifier |
source |
str |
"canvas" or "eval" |
step |
int |
GraphExecutor step counter |
node_id |
str |
Node instance ID from graph |
node_type |
str |
Node type (e.g. env_habitat__observe) |
node_label |
str |
Human-readable label |
duration_ms |
float |
Execution time in milliseconds |
inputs |
dict |
Serialized port inputs (pre-execution) |
outputs |
dict |
Serialized port outputs (post-execution) |
inner_log |
list[dict] |
Voluntary _self_log() entries |
port_wire_types |
dict[str, str] |
Port name โ wire type mapping |
parent_node_id |
str | None |
Dynamic Fire-List provenance โ the spawner node id for an ephemeral DynamicFireListNode child (logging/models.py ยท NodeLogEntry, populated at graph_executor.py ยท _fire_node) |
dynamic_index |
int | None |
Child index within the spawned fire-list (logging/models.py ยท NodeLogEntry) |
error |
str | None |
Error message if node failed |
Both provenance fields are present on every JSONL line (null for ordinary, non-spawned firings).
ExecutionSummary โ aggregate stats:
| Field | Type | Description |
|---|---|---|
execution_id |
str |
Run identifier |
source |
str |
"canvas" or "eval" |
started_at / ended_at |
datetime |
Time bounds |
total_steps |
int |
Max step counter reached |
total_firings |
int |
Total node firings |
error_count |
int |
Firings that raised exceptions |
node_types_fired |
list[str] |
Distinct node types that fired |
2.2 Ring Buffer + Write Queue
- Ring buffer:
collections.deque(maxlen=2000)โ thread-safe, fixed-size window for real-time queries via REST - Write queue: accumulates entries during an iteration;
flush()batch-writes them to JSONL atiterOutboundaries (not per-node) - Final flush: on
GraphCompleteorGraphError, the executor callsflush()one last time
2.3 Key Methods
| Method | Purpose |
|---|---|
log_node(step, node_id, ..., inner_log, port_wire_types, error, parent_node_id, dynamic_index) |
Record one firing โ ring buffer + write queue |
flush() |
Batch-append write queue to JSONL, clear queue |
get_entries(node_id?, node_type?, step?, limit, offset) |
Query ring buffer with optional filters |
get_summary() |
Return ExecutionSummary with aggregate stats |
3. Serialization & Asset Storage
Large values (images, tensors, long strings) cannot go into JSONL as-is. Two serialization functions handle this.
3.1 log_serialize()
Recursive serializer that normalizes non-JSON values into JSON-safe forms. Strings are persisted in full โ there is no length cap or truncation (logger.py ยท log_serialize; module docstring: "Strings are persisted in full"):
| Input type | Output |
|---|---|
str (any length) |
the string verbatim โ full LLM prompts/responses survive untruncated |
bytes |
{"__type": "bytes", "length": N} |
| numpy array / torch tensor | {"__type": "ndarray", "shape": [H,W,C], "dtype": "uint8"} |
| Recursion > depth 5 | "<max_depth>" |
| Unknown type | "<ClassName>" |
The max_str_len parameter still exists in the signature (and inner_log passes 50000) but is inert โ no value of it triggers truncation. Image/array bytes are the only payloads diverted out of the JSONL, via save_asset() (ยง3.2).
3.2 log_serialize_with_assets()
Wire-type-aware serializer that saves images as sidecar files instead of embedding them:
| Wire type | Behavior |
|---|---|
LIST[T] |
Emit a compact count marker {"__type": "list", "wire_type": "LIST[T]", "count": N} โ no per-tile assets to avoid N-image bloat |
IMAGE |
Save RGB numpy array as JPEG (quality 85) โ assets/s{step}_f{idx}_{label}__{port}.jpg |
DEPTH |
Normalize depth to uint8, save as PNG โ assets/s{step}_f{idx}_{label}__{port}.png |
OBSERVATION |
Recurse into rgb and depth sub-keys, save each as above |
STEP_RESULT |
Recurse into observation sub-key, then handle as OBSERVATION |
| Everything else | Fall through to log_serialize() |
Saved assets are referenced in the JSONL as:
{ "__type": "asset", "path": "assets/s0_f0_Habitat__Observe__rgb.jpg", "wire_type": "IMAGE", "shape": [480, 640, 3], "dtype": "uint8" }
Composite observations produce grouped references:
{ "__type": "asset_group", "wire_type": "OBSERVATION", "rgb": {"__type": "asset", "path": "assets/s0_f0_Habitat__Observe__obs_rgb.jpg", ...}, "depth": {"__type": "asset", "path": "assets/s0_f0_Habitat__Observe__obs_depth.png", ...} }
4. Executor Integration
4.1 Logger Creation
LoopRunner (app/agent_loop/loop_runner.py) creates the logger at run start โ but only when one was not injected. Eval passes in its own per-episode logger (next paragraph); the canvas branch below runs only when logger is None and self._execution_id (loop_runner.py ยท run):
persist_dir = os.path.join(repo_root, "outputs", "runs", self._execution_id) logger = ExecutionLogger( execution_id=self._execution_id, source="canvas", persist_dir=persist_dir, )
For eval runs, the source is "eval" and the logger is created per episode inside BatchEvalRunner._run_one_episode (agent_loop/eval_batch.py) with persist_dir = outputs/eval_runs/{run_id}/episodes/ep{idx:04d}/ and execution_id = "{run_id}_ep{idx:04d}". Each episode owns its own log.jsonl + assets/ โ no interleaving across workers (see ADR-eval-004).
4.2 Node Firing Instrumentation
GraphExecutor._fire_node() instruments every node firing:
- Instantiate the handler (
BaseCanvasNodesubclass) and injectpending_inputs+ persistent state. - Run the
PreNodeExecutehook, then stampt0 = time.monotonic(). - Execute the node:
result = await instance.forward(inputs, ctx). - Compute
elapsed_ms, then run thePostNodeExecutehook. - Collect the voluntary log:
inner_log = instance.log(). logger.log_node(step, node_id, โฆ, inner_log, port_wire_types, error)โ records exterior + interior layers in one entry.- Broadcast the
exec_logWS event (unless suppressed โ eval suppresses it).
4.4 Per-node LLM usage hook (ADR-observability-005)
Token usage and dollar cost are tracked per node firing without nodesets having to plumb them. The executor wraps every node.forward() with a ContextVar-scoped accumulator (graph_executor.py ยท _fire_node):
usage_bucket: dict = {} usage_token = _current_node_usage.set(usage_bucket) # standard/telemetry.py ยท _current_node_usage try: result = await node.forward(inputs, ctx) finally: _current_node_usage.reset(usage_token)
Inside the forward(), every completion round trip โ app.llm.complete, which the llm_complete / llm_complete_n / vlm_complete / vlm_complete_n wrappers all reach โ adds the usage its transport reported (response.usage + litellm.completion_cost() for the litellm adapter) into the bucket (complete.py ยท _accumulate_usage). The executor then emits one log entry per node with the aggregate attached:
| Field | Source |
|---|---|
calls | increments per transport round trip (a native-n call counts once; each single-sample fill counts once) |
prompt_tokens / completion_tokens / total_tokens | response.usage on each provider response |
cached_tokens | response.usage.prompt_tokens_details.cached_tokens when present |
usd_cost | litellm.completion_cost(response); 0.0 for models litellm doesn't recognise (local Ollama, custom vLLM) |
model | first non-empty response.model |
Properties: voluntary (nodes that don't call the LLM helpers get an empty bucket), per firing (multi-scope graphs and iterated nodes get one entry per call site), and ContextVar-propagating (survives the internal await chain). Server-mode nodesets capture usage in the subprocess (server/invoke.py sets the same ContextVar around the hosted handler) and return it in the /call reply's usage field; the proxy node folds it into the firing's bucket (proxy.py ยท _make_forward โ telemetry.merge_usage), and the hosted node's _self_log entries ride back the same way (log field). See LLM Config ยง12 for the full contract.
4.3 Flush Points
The logger's write queue is flushed at three points:
| When | Where | Why |
|---|---|---|
iterOut firing |
graph_executor.py ยท run |
Batch-write at iteration boundary |
GraphComplete |
graph_executor.py ยท run |
Final flush on success |
GraphError |
graph_executor.py ยท run |
Flush even on failure (inside the except block, after the GraphError hook) |
5. REST API
Five query endpoints + one asset-serving endpoint in app/api/execution/logs.py, mounted at /api/logs. A live execution is answered from its in-memory ExecutionLogger; every other answer comes from the LogStore (app/logging/store.py), the one place that knows the on-disk layout โ canvas_runs/{execution_id}/log.jsonl and eval_runs/{run_id}/episodes/ep{idx:04d}/log.jsonl, an eval episode addressed by the composite id {run_id}_ep{idx:04d} (LogStore.locate) โ plus the tolerant JSONL reader (read_entries) and the summarize / timing aggregations. The replay API and the Monitor's run view (services/system_runs.py) read through the same store, so a layout change is one edit.
| Endpoint | Method | Purpose |
|---|---|---|
/api/logs |
GET | List recent executions (canvas + eval, sorted by mtime, limit 100) |
/api/logs/{execution_id} |
GET | Log entries with filters (node_id, node_type, step, limit, offset) |
/api/logs/{execution_id}/node/{node_id} |
GET | One node's output history across all steps |
/api/logs/{execution_id}/summary |
GET | Execution summary (total firings, errors, node types) |
/api/logs/{execution_id}/graph |
GET | Graph definition snapshot (graph.json) saved at run start for canvas replay |
/api/logs/{execution_id}/assets/{filename} |
GET | Serve sidecar asset files (JPEG/PNG) with correct media types |
Query priority: active in-memory logger is checked first; historical JSONL files are the fallback.
Security: _sanitize_id() prevents path traversal attacks on execution_id and filename parameters.
6. WebSocket Events
After each node firing, the executor broadcasts a lightweight exec_log event via WebSocket:
interface ExecLogEvent { step: number; node_id: string; node_type: string; node_label: string; duration_ms: number; error: string | null; has_inner_log: boolean; }
- No inputs/outputs in the WS payload โ only metadata. Full data is fetched via REST on demand.
- Suppressed in eval mode when
principles.suppress_nav_events = True(avoids flooding during batch evaluation).
7. Frontend: Log Viewer
7.1 Three View Modes
The Log Viewer supports three view modes, toggled via a segmented control in the filter bar:
| Mode | Description |
|---|---|
| Overall (default) | Compact virtualized rows with 24x24 inline thumbnails. Click a row to open a side detail pane (master-detail layout). |
| Detail | Every row auto-expands inline showing full outputs, inputs, inner_log, and large images (up to 320px). Each entry renders as a card with visible borders. No side pane. |
| Canvas | Read-only React Flow canvas rendering the executed graph with step-by-step replay. Node status highlighting (firing/completed/error). Resizable detail panel below. |
7.2 Page Structure
The Log Viewer is a full-page inspector (LogViewerPage.tsx) with three areas:
| Component | File | Role |
|---|---|---|
LogViewerPage |
LogViewerPage.tsx |
Layout: sidebar + main content, viewMode state |
ExecutionList |
ExecutionList.tsx |
Sidebar: lists recent runs with size/mtime, highlights active execution |
LogSummaryBar |
LogSummaryBar.tsx |
Toolbar: total firings, errors, node types, live/historical badge |
LogFilters |
LogFilters.tsx |
Search box, node type dropdown, errors-only toggle, 3-way mode toggle |
LogEntryList |
LogEntryList.tsx |
Virtualized list for overall/detail modes with @tanstack/react-virtual |
LogCanvasView |
LogCanvasView.tsx |
Canvas replay: read-only React Flow + step navigation + resizable detail panel |
LogContext |
LogContext.tsx |
React context providing executionId and viewMode to renderers |
7.3 Canvas Replay Mode
LogCanvasView renders the executed graph as a read-only React Flow canvas. It requires graph.json (saved by the backend at execution start).
Data loading: fetches graph.json, log entries, and node schemas in parallel at mount.
Node rendering: uses the same proxiedNodeTypes, toFlowNodes(), and toFlowEdges() as the real canvas editor โ visual parity with the edit canvas.
Node status highlighting via style overrides:
| Status | Visual |
|---|---|
idle |
Default node appearance |
firing |
Blue outline + glow (box-shadow) |
completed |
Subtle green outline |
error |
Red outline |
Step controls: First / Prev / Play-Pause / Next / Last. Auto-play at 800ms interval. Step indicator shows firing index, step number, node label, type, and duration.
Resizable detail panel: drag handle between canvas and panel, 80pxโ600px range.
7.4 Value Renderer Registry
The renderer registry (renderers/registry.ts) dispatches values to specialized React components based on a 4-step resolution:
- Explicit wire type from
port_wire_types[key]โ use that wire type's renderer __typemarker fromlog_serialize()(e.g."asset","ndarray","bytes") โ use__type's renderertypeofprimitive (string,number,boolean) โ use primitive renderer- Fallback โ
JsonTreeRenderer
| Renderer | Handles | Wire Types / Markers |
|---|---|---|
ImageRenderer |
Asset images, composite observation groups | IMAGE, DEPTH, OBSERVATION, asset, asset_group |
TextRenderer |
Text output (strings are stored in full) | TEXT |
ActionRenderer |
Formatted navigation actions | ACTION |
MetricsRenderer |
Key-value metric tables | METRICS |
StepResultRenderer |
Nested step results with observation sub-groups | STEP_RESULT |
TensorRenderer |
Shape/dtype summaries for arrays and tensors | ndarray, Tensor, bytes |
ErrorRenderer |
Highlighted error messages | (used inline for error field) |
PrimitiveRenderer |
Scalars, booleans, null | BOOL, string, number, boolean |
JsonTreeRenderer |
Collapsible tree for complex objects | STATE, ANY, fallback |
All renderers are lazy-loaded (React.lazy) for bundle splitting.
7.5 Canvas Integration
- LogPanel (
canvas/panels/LogPanel.tsx): inline log viewer inside the OutputDrawer's Logs tab, subscribes toexec_logWS events for live updates - StepLogPanel (
canvas/panels/StepLogPanel.tsx): action/thinking log side panel with ActionLog + LLMLog views
7.6 API Client
logApi.ts provides typed fetch wrappers:
listExecutions(): Promise<ExecutionListItem[]> getEntries(id, filters?): Promise<LogEntry[]> getNodeHistory(id, nodeId): Promise<LogEntry[]> getSummary(id): Promise<ExecutionSummary> assetUrl(id, path): string // builds URL for <img src>
7.7 Node Color Coding
nodeColors.ts maps node type prefixes to Tailwind color classes for visual categorization in the log list (e.g. env_habitat__* โ blue, llm__* โ purple).
8. Storage Layout
outputs/runs/โ canvas executionsexec_<id>/graph.jsonโ GraphDefinition snapshot for canvas replaylog.jsonlโ oneNodeLogEntryper lineassets/โ sidecar image files, e.g.s0_f0_Habitat__Observe__rgb.jpg/โฆ__depth.png
eval_runs/โ eval executions (ADR-eval-004){run_id}/โ timestamp + collision suffix, e.g.20260515_143052spec.json,shared_urls.jsonโ input contractsummary.json,graph.jsonโ run-level snapshotstdout.log,stderr.log,_DONEepisodes/ep{idx:04d}/โ one self-contained dir per episodelog.jsonlโ stampedexecution_id = {run_id}_ep{idx:04d}assets/โ this episode's image artefactsepisode.jsonโ self-describing row fromsummary.json
JSONL format โ each line is one NodeLogEntry:
{ "timestamp": "2026-04-07T14:12:41.757840", "execution_id": "exec_1775571161675_2z1w47c", "source": "canvas", "step": 0, "node_id": "episode_info", "node_type": "env_habitat__episode_info", "node_label": "Habitat: Episode Info", "duration_ms": 25.43, "inputs": {}, "outputs": {"instruction": "Walk through the door...", "episode_id": "1"}, "inner_log": [{"key": "format_length", "value": 640}], "port_wire_types": {"instruction": "TEXT", "episode_id": "TEXT"}, "parent_node_id": null, "dynamic_index": null, "error": null }
When a node makes one or more LLM/VLM calls, the per-node usage bucket (ยง4.4) is appended to that node's inner_log as a {"key": "usage", "value": {calls, prompt_tokens, completion_tokens, total_tokens, cached_tokens, usd_cost, model}} entry โ only when calls > 0 (graph_executor.py ยท _fire_node). It rides inside inner_log, not as a top-level field.
9. File Manifest
9.1 Backend
| File | Lines | Role |
|---|---|---|
app/logging/logger.py |
402 | ExecutionLogger, log_serialize(), log_serialize_with_assets(), save_asset() |
app/logging/models.py |
50 | NodeLogEntry, ExecutionSummary Pydantic models |
app/logging/__init__.py |
8 | Public exports |
app/logging/store.py |
365 | LogStore โ layout, addressing (locate / episode_execution_id), read_entries, list_executions, summarize, timing; behind the logs API, the replay API and system_runs |
app/api/execution/logs.py |
172 | 6 REST endpoints for log queries, graph retrieval, and asset serving โ live runs from the logger, everything else through LogStore |
app/components/bases.py |
554 | BaseCanvasNode._self_log() and .log() voluntary logging API |
app/agent_loop/graph_executor.py |
~1860 | Logger integration: log_node() after each firing, flush() at boundaries |
app/agent_loop/loop_runner.py |
216 | Logger instantiation with persist_dir (only when none is injected) |
9.2 Frontend
| File | Lines | Role |
|---|---|---|
src/logs/types.ts |
63 | TypeScript interfaces (LogEntry, ExecLogEvent, AssetRef, etc.) |
src/logs/logApi.ts |
54 | REST client (4 query functions + asset URL builder) |
src/logs/LogContext.tsx |
4 | React context for executionId |
src/logs/LogViewerPage.tsx |
95 | Full-page log inspector layout with viewMode state |
src/logs/LogCanvasView.tsx |
290 | Canvas replay: read-only React Flow + step nav + resizable detail panel |
src/logs/ExecutionList.tsx |
86 | Sidebar: recent execution list |
src/logs/LogSummaryBar.tsx |
83 | Toolbar: aggregate stats |
src/logs/LogFilters.tsx |
75 | Search / filter controls |
src/logs/LogEntryList.tsx |
449 | Virtualized master-detail entry list |
src/logs/nodeColors.ts |
54 | Node type โ color mapping |
src/logs/renderers/registry.ts |
84 | Wire-type-aware renderer dispatch |
src/logs/renderers/*.tsx |
~9 files | Specialized value renderers (Image, Text, Action, Metrics, etc.) |
src/canvas/panels/LogPanel.tsx |
โ | Canvas OutputDrawer inline log (WS subscription) |