LLM/VLM Configuration System

The LLM/VLM configuration system lets any canvas node call any of 24 providers — OpenAI, Anthropic, Google, Ollama, and 20+ more — through one litellm-backed async API. A profile names a provider + model; API keys never live in the profile, they're read from the provider's standard env var at call time. Each node can pick its own profile, so a single graph can reason with GPT-4o, caption with Claude, and run cheap inference on local Ollama. Designed in ADR-platform-003, hardened in ADR-platform-006.

How a node gets its model — the whole call path. (§3 zooms into the resolve config step; §12 into the usage hook.)

1. What it is, and the one rule

Canvas nodes — chiefly LLMCallNode — need to reach an LLM/VLM without hard-coding a provider SDK. This system is that indirection: litellm is the single abstraction layer (no per-provider SDKs; httpx appears only in key_validator.py for the Ollama health check). Four design choices shape it:

• The one rule — no secrets on disk. profiles.json carries only provider / model / base_url / api_type and is git-safe; keys live in per-provider env vars (OPENAI_API_KEY, …) read at call time (ADR-platform-006).
• Field-level fallback — each config field (key, model, base_url, api_type) resolves independently: the profile's value, else the provider registry default.
• Hot reload — mtime-based cache invalidation, so a CLI edit is picked up by a running server within ~2 s.
• Graceful degradation — no active profile, or the provider env var unset, drops to mock mode (get_llm_config() returns None, the node renders a placeholder) — unless AGENTCANVAS_STRICT_ERRORS is set (§11.5).

2. The smallest setup

Two steps put a live model behind every llmCall node — set the provider's env var, then create and activate a profile:

# 1. the key lives in the environment, never in a file
export OPENAI_API_KEY=sk-...

# 2. create a profile (provider + model) and make it the active default
python -m agentcanvas.backend.app config set openai --provider openai --model gpt-4o
python -m agentcanvas.backend.app config activate openai

Now an llmCall node with a blank profile field resolves to the active openai profile; a node that sets profile = "anthropic" uses that one instead (§9). No key ever touches profiles.json.

3. The resolution chain at a glance

Every LLM call resolves its config in two stages: (1) load the named (or active) profile from profiles.json, (2) read the API key from the provider's standard env var. ADR-platform-006 removed the previous AGENTCANVAS_* / VLM_* global env-var fallback chain — keys are now per provider, never on disk.

The merge of profile fields + registry defaults happens inside resolve_provider_config() in providers.py: empty base_url / api_type / model on the profile inherit from PROVIDER_REGISTRY[provider]. litellm_prefix always comes from the registry (not user-overridable). If api_key ends up empty and the provider is not Ollama, get_llm_config() returns None (mock mode).

Special cases:

• Ollama has env_var="" in the registry and skips the key check (local server)
• Custom provider is the only entry whose env var is AGENTCANVAS_API_KEY (general-purpose self-hosted endpoints)
• No active profile + no explicit profile_name returns None immediately (no env-only mode)
• Explicit profile_name (from per-node selection) skips the active-profile lookup

3.1 Resolution Code Path

Inside a node's forward(), one call resolves everything — get_llm_config(profile_name) (call.py · get_llm_config):

1. ProfileStore.get(name) — disk read with an mtime cache (TTL 2 s).
2. resolve_provider_config(profile) — merge profile + registry defaults, calling get_provider_api_key(provider) = os.environ[ProviderDef.env_var].
3. If api_key == "" and api_type != "ollama" → return None (mock mode).
4. Otherwise → LLMConfig(api_key, base_url, model, api_type, litellm_prefix).

Source: call.py · get_llm_config, providers.py · get_provider_api_key, providers.py · resolve_provider_config.

4. Provider Registry

providers.py defines 24 pre-configured providers. Each entry supplies default base_url, api_type, and default_model, so users only need to provide an API key.

4.1 Supported Providers

Total: 24 providers (count confirmed in PROVIDER_REGISTRY in app/llm/providers.py).

4.2 Registry Entry Structure

@dataclass(frozen=True)
class ProviderDef:
    label: str            # Human-readable name (e.g. "OpenAI")
    base_url: str         # Default API endpoint
    api_type: str         # Protocol: "openai" | "anthropic" | "google" | "ollama"
    default_model: str    # Suggested model (e.g. "gpt-4o")
    litellm_prefix: str   # litellm model prefix (e.g. "openai", "gemini", "anthropic")
    env_var: str          # Standard env-var name for the API key (e.g. "OPENAI_API_KEY")
                          # Empty ("") for Ollama — local server, no key needed

When creating a profile with provider="openai", the registry fills in https://api.openai.com/v1, openai, and gpt-4o automatically. The profile can override base_url / api_type / model; litellm_prefix and env_var always come from the registry. The runtime reads os.environ[env_var] at call time (ADR-platform-006) — there is no on-disk slot for the key.

Source: app/llm/providers.py

5. Profile Store

Profiles are persisted in agentcanvas/backend/profiles.json (checked in alongside requirements.txt; override location via AGENTCANVAS_PROFILES_FILE=<absolute-path>). The ProfileStore class manages CRUD with thread-safe locking and mtime-based cache invalidation. The file is git-safe by construction (no secrets) — see ADR-platform-006.

5.1 File Format

{
  "schema_version": 1,
  "active": "openai",
  "profiles": {
    "openai": {
      "provider": "openai",
      "model": "gpt-4o",
      "base_url": "",
      "api_type": ""
    },
    "anthropic": {
      "provider": "anthropic",
      "model": "claude-sonnet-4-6",
      "base_url": "",
      "api_type": ""
    }
  }
}

• No api_key field — keys live in env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, …). Legacy api_key fields in older files are silently dropped on load and never written back (llm/profiles.py · ProfileStore.load)
• active: name of the default profile used when no profile is specified
• Empty base_url / api_type fields fall back to registry defaults; empty model falls back to ProviderDef.default_model
• File written with default umask permissions — no chmod 0o600, since the file carries no secrets (safe for shared checkouts and git)

5.2 Cache Invalidation

ProfileStore avoids re-reading the file on every request:

1. In-memory cache of ProfileData
2. mtime check gated to at most once every 2 seconds (_mtime_check_interval)
3. If file mtime changed, cache is invalidated and re-read on next access

This means CLI changes or manual file edits are picked up by the running server within ~2 seconds, with no restart required.

5.3 Profile Data Model

@dataclass
class LLMProfile:
    """Non-secret per-profile configuration. API keys are NEVER stored here."""
    provider: str      # Key into PROVIDER_REGISTRY (e.g. "openai")
    model: str         # Model name (e.g. "gpt-4o")
    base_url: str = "" # "" = use registry default
    api_type: str = "" # "" = use registry default

LLMProfile is the on-disk shape (no api_key field — ADR-platform-006). At call time, resolve_provider_config() merges it with registry defaults, looks up the env-var key via get_provider_api_key(provider), and resolves litellm_prefix, producing an LLMConfig:

@dataclass
class LLMConfig:
    api_key: str
    base_url: str
    model: str
    api_type: str         # "openai" | "anthropic" | "google" | "ollama"
    litellm_prefix: str   # litellm model prefix (e.g. "openai", "gemini", "ollama_chat")

Source: app/llm/profiles.py, app/llm/call.py

6. Environment Variables

API keys are the only piece of config that comes from env vars — and they are per provider, named after each vendor's de-facto SDK convention. Profile metadata (provider / model / base_url / api_type) lives in profiles.json; there is no global AGENTCANVAS_* or VLM_* env-var fallback for those fields any more (removed in ADR-platform-006).

6.1 Per-Provider Env Vars

Each entry in PROVIDER_REGISTRY declares an env_var field — the env var the runtime reads for that provider's key. Common ones:

Full mapping in providers.py · PROVIDER_REGISTRY. To see what's set on a given machine: python -m agentcanvas.backend.app config env.

6.2 Configuring a Provider

One-liner for a typical setup:

# 1. Create the profile (no key, just metadata)
python -m agentcanvas.backend.app config set openai --provider openai --model gpt-4o

# 2. Export the key in your shell (e.g. ~/.bashrc)
export OPENAI_API_KEY=sk-...

# 3. Activate the profile
python -m agentcanvas.backend.app config activate openai

Note: agentcanvas/backend/.env.example in the repo still shows the legacy AGENTCANVAS_API_KEY/MODEL/BASE_URL/API_TYPE and VLM_* envs. Those credential/model names are dead — never read by call.py — so use the per-provider names above instead. (Two unrelated AGENTCANVAS_* vars are still live and are not credentials: AGENTCANVAS_STRICT_ERRORS (§11.5) and AGENTCANVAS_PROFILES_FILE (profile-store path override). File is queued for cleanup.)

6.3 Mock Mode

get_llm_config() returns None whenever it can't resolve a usable config: no active profile, profile name unknown, or the provider's env var is unset (Ollama excepted). Calling nodes treat None as a mock — they emit a placeholder string and continue, so a graph won't crash for missing keys; it will just produce mock outputs visibly in the canvas.

7. CLI Tool

A standalone CLI for profile management that works without the FastAPI server running. Changes are picked up by the running server via mtime invalidation.

7.1 Commands

8 commands total. All accept --json for machine-readable output; show and env also accept --show-keys to unmask values pulled from env vars.

# Entry point (two forms, both work)
python -m agentcanvas.backend.app.cli config <command>
python -m agentcanvas.backend.app config <command>

# List all profiles
config list [--json]

# Show a single profile (resolved against env)
config show <name> [--show-keys] [--json]

# Create or update a profile (metadata only — no --api-key flag)
config set <name> [--provider <id>] [--model <model>] [--base-url <url>] [--api-type <type>]
# On create, prints: "Set the API key via env: export OPENAI_API_KEY=<your-key>"

# Delete a profile
config delete <name>

# Set the active (default) profile; omit name to clear
config activate [<name>]

# Test API key connectivity (reads key from env var, calls litellm validate)
config test [<name>]

# List all 24 supported providers (with their env_var names)
config providers [--json]

# Show which provider env vars are set on this machine + active profile
config env [--show-keys] [--json]

API-key entry is intentionally not a CLI step (no --api-key flag, no getpass prompt). The key is read from the provider's env var (OPENAI_API_KEY, …) at call time — set it in your shell profile or via your secret manager, then run any command above. ADR-platform-006 keeps the CLI write path entirely free of secrets.

7.2 Example Workflow

# 1. Export keys in your shell (~/.bashrc / ~/.zshrc / .envrc)
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# 2. Create profile metadata (one per provider/model pair you want to pick from)
python -m agentcanvas.backend.app config set openai --provider openai --model gpt-4o
python -m agentcanvas.backend.app config set anthropic --provider anthropic --model claude-sonnet-4-6
python -m agentcanvas.backend.app config set ollama --provider ollama --model llama3.2

# 3. Activate OpenAI as default for nodes that don't specify a profile
python -m agentcanvas.backend.app config activate openai

# 4. Verify the key actually works (round-trips to the provider)
python -m agentcanvas.backend.app config test

# 5. Sanity check which env vars are picked up
python -m agentcanvas.backend.app config env

Source: app/llm/cli.py

8. REST API

Profile management is exposed via FastAPI endpoints under /api/profiles.

8.1 Endpoints

8.2 Batch Upsert

The frontend sends a single PUT /api/profiles/batch call with all changes:

{
  "keys": { "openai": "sk-...", "anthropic": "sk-ant-..." },
  "active": "openai",
  "active_model": "gpt-4o",
  "overrides": {
    "ollama": { "base_url": "http://localhost:11434", "model": "llama3.2" },
    "custom": { "base_url": "https://my-api.com/v1", "model": "my-model", "api_type": "openai" }
  }
}

keys is accepted but ignored (kept in the request shape only for backward compat with older frontends — see BatchUpsertBody docstring in api/profiles.py · BatchUpsertBody). Real API keys live in env vars. The endpoint does create empty profile metadata for each provider listed in keys so the UI's per-provider config slot exists, applies overrides to base_url/model/api_type, and sets the active profile — all atomically.

8.3 Security

• API keys are never in profiles.json — they live in per-provider env vars (ADR-platform-006). GET responses expose only api_key_set: true/false, derived from os.environ[env_var] presence at request time
• profiles.json is written with default umask permissions and is git-safe
• Frontend Settings Modal still renders a key input field (for legacy frontends), but the value is dropped server-side; the right place to set keys is the shell env, not the modal

Source: app/api/platform/profiles.py

9. Per-Node Model Selection

Individual canvas nodes can override the default profile. LLMCallNode is the only node that carries a profile config field (builtin_nodes.py · LLMCallNode):

ConfigField("profile", "select", "Model", default="",
            options=[{"value": "__DYNAMIC_PROFILES__", "label": ""}])

The __DYNAMIC_PROFILES__ sentinel in options is the load-bearing part: the frontend swaps it for the live list of configured profiles when it renders the dropdown. There is no static option list and no description kwarg.

At execution time:

profile_name = self.config.get("profile", "")
llm_config = get_llm_config(profile_name)
# "" → active profile → env fallback → None (mock)
# "anthropic" → that specific profile

This enables mixed-model graphs: one node reasons with GPT-4o, another captions with Claude, a third does cheap inference with local Ollama.

The frontend renders this as a dropdown in the node's config panel. The dropdown is populated from the list of configured profiles (providers with saved keys + Ollama). A profiles-changed window event refreshes the dropdown when settings are saved.

Source: app/agent_loop/builtin_nodes.py (search for profile ConfigField). The legacy app/agent_loop/graph_executor.py referenced in older docs has been removed; node handlers live in builtin_nodes.py and the executor in graph_executor.py.

10. Frontend Settings Modal

SettingsModal.tsx provides the GUI configuration surface.

10.1 Layout

10.2 Behavior

• Top 5 providers always visible: OpenAI, Anthropic, Google, DeepSeek, Ollama
• More providers collapsible section with 19 additional providers
• Custom provider section for arbitrary endpoints
• Star button sets the default model (active profile)
• Per-provider model override inline text field next to each configured model
• On save: PUT /api/profiles/batch with all changes, then window.dispatchEvent(new Event('profiles-changed')) to refresh node dropdowns

Source: agentcanvas/frontend/src/components/SettingsModal.tsx

11. API Call Protocols

app/llm/call.py uses litellm.acompletion (no individual provider SDKs). Both text-only (llm_complete) and multimodal (vlm_complete) variants are supported, each with a multi-sample *_n counterpart (llm_complete_n / vlm_complete_n) for the OpenAI n parameter. litellm translates the unified OpenAI-format message structure to provider-native wire formats internally.

11.1 Protocol Dispatch

api_type (from LLMConfig) selects the litellm model prefix via API_TYPE_TO_LITELLM_PREFIX:

The litellm model string is built as {prefix}/{model} and passed to litellm.acompletion(). litellm handles auth headers, endpoint routing, and protocol translation.

11.2 Image Handling (vlm_complete)

vlm_complete takes a list of base64-encoded PNG/JPEG strings and builds an OpenAI-format multimodal content array (image_url blocks). litellm translates these to provider-native formats (Anthropic source.type:base64, Google inline_data, Ollama images array) automatically.

An optional image_labels list can interleave per-image text captions before each image (used by MapGPT's gpt_infer interleave shape).

An optional detail argument ("low" / "high" / "auto", default "low") is forwarded into each image_url block as OpenAI's resolution hint: "low" caps every image at a single 512px tile (token-cheap), "high" enables multi-tile high-resolution analysis for fine spatial grounding. vlm_complete_n threads the same hint to every candidate. The builtin llmCall node surfaces this as a second config field — image_detail (select: low / high / auto, default low, builtin_nodes.py · LLMCallNode) — so a node whose vision task needs fine grounding (e.g. a viewpoint-picking navigator) can be set to high without touching code.

If images is empty, vlm_complete falls back to llm_complete (text-only path).

Multi-sampling. vlm_complete_n(..., n) returns up to n candidate completions — the vision counterpart to llm_complete_n, used for self-consistency / multi-candidate planning. It first attempts provider-native n (the image prompt is then billed once for all samples); providers that ignore n for vision requests — notably Anthropic — collapse the response to a single choice, which is detected and the shortfall is filled with concurrent single-sample vlm_complete calls. The builtin llmCall node routes its image path through vlm_complete_n whenever config.n > 1 and exposes the candidates on the responses port.

11.3 Timeouts

11.4 Public API

from app.llm import get_llm_config, llm_complete, vlm_complete

config = get_llm_config("openai")  # or "" for active profile

# Text-only
response = await llm_complete(config, messages, system_prompt="...", max_tokens=1024)

# Multimodal (text + list of base64 images, optional per-image labels)
response = await vlm_complete(
    config,
    prompt="Describe these images",
    images=[base64_png_1, base64_png_2],
    image_labels=["Front view", "Side view"],  # optional
)

Temperature defaults are asymmetric and unstated at the call site. llm_complete / llm_complete_n default temperature=0.7; vlm_complete / vlm_complete_n default temperature=0.3 (vision is run cooler). max_tokens has no default — pass it explicitly. Both are forwarded verbatim to litellm.acompletion with no per-model adjustment (see §11.5).

Source: app/llm/call.py

11.5 Strict errors & the gpt-5 sharp edge

By default every helper degrades gracefully: a litellm exception, timeout, or unparseable response makes llm_complete return None and the *_n variants return [], so a misconfigured node renders a placeholder rather than crashing the run. Setting the env var AGENTCANVAS_STRICT_ERRORS (to 1/true/yes/on, read fresh on every call via call.py · _strict_errors_enabled — deliberately uncached, so env-worker subprocesses inherit the backend's launch-time setting) flips this: every helper re-raises instead. Eval and smoke runs set it so a silent empty response can't masquerade as a completed episode.

12. Per-Node Usage Hook (ADR-observability-005)

Token usage and dollar cost are tracked per node firing, not per nodeset and not globally. Nodesets never need to plumb usage manually — the wiring is one ContextVar in call.py and one set/reset pair around forward() in the executor.

# call.py · _current_node_usage
_current_node_usage: contextvars.ContextVar[dict | None] = contextvars.ContextVar(
    "_current_node_usage", default=None
)

# Every llm_complete / llm_complete_n / vlm_complete call inside a forward()
# adds its response usage into this bucket via _accumulate_usage():
#   bucket["calls"]              += 1
#   bucket["prompt_tokens"]      += response.usage.prompt_tokens
#   bucket["completion_tokens"]  += response.usage.completion_tokens
#   bucket["total_tokens"]       += response.usage.total_tokens
#   bucket["cached_tokens"]      += response.usage.prompt_tokens_details.cached_tokens
#   bucket["usd_cost"]           += litellm.completion_cost(response)
#   bucket["model"]               = first non-empty response.model

# graph_executor.py · _fire_node
usage_bucket: dict = {}
usage_token = _current_node_usage.set(usage_bucket)
try:
    result = await node.forward(inputs, ctx)
finally:
    _current_node_usage.reset(usage_token)
# bucket now holds aggregate usage for this firing; executor emits one
# log entry per node with the bucket attached.

Properties of this design:

• Voluntary, not invasive — nodes that don't call llm_complete get an empty bucket; nodes that call N times get one aggregated entry, not N events
• Per firing, not per node lifetime — multi-scope graphs and iterated nodes accumulate per call site, exactly what you want for cost reports
• Works inside subprocesses — ContextVars propagate into asyncio tasks; the bucket survives any internal await chain. Server-mode nodesets (AutoServerApp) capture usage in the subprocess and surface it on the response, so the framework can still attribute it to the proxy node
• Cost may be zero for models litellm doesn't recognise (local Ollama, custom vLLM endpoints) — usage tokens still counted, usd_cost stays 0.0

Source: call.py · _current_node_usage + call.py · _accumulate_usage (bucket + accumulator), graph_executor.py · _fire_node (set/reset around forward()), call.py · _extract_usage (response → usage dict).

13. Key Files