Basic Agent NodeSet

General-purpose tools for LLM-based VLN agents, providing 11 canvas nodes across 5 categories: scratch pad (working memory), web grounding, vision, spatial math, and episode context. Use these nodes to build agents that can read/write notes, search the web, analyze images, compute distances and headings, and access episode metadata.

1. Overview

The Basic Agent NodeSet (basic_agent_tools) provides a foundational toolkit for agents that reason about VLN tasks. The 11 nodes are organized into 5 functional categories:

Each node is registered in the canvas component system and can be wired into agent graphs via the canvas UI.

2. Canvas Nodes

2.1 Scratch Pad

The scratch pad nodes provide key-value storage for LLM working memory (planning, reasoning state, intermediate results). Storage is shared across all nodes in the same pad_id.

UI Color: Amber

Example workflow:

[LLM Node] → key="plan" → [Scratch Pad: Write]
                        → content="explore north hallway"
[Scratch Pad: List] → keys=["plan"]
[Scratch Pad: Read] → key="plan" → content="explore north hallway"

2.2 Web Grounding

The web grounding nodes enable agents to search and fetch information from the public web.

UI Color: Sky

Implementation details: - Web: Search uses DuckDuckGo's HTML-lite interface with a standard User-Agent header. Handles network failures gracefully. - Web: Fetch follows redirects (max 3), strips <script> and <style> blocks, and truncates output if it exceeds max_chars config.

Example workflow:

[LLM: "search for tall buildings"] → [Web: Search] → results="1. Empire State..."
                                   → [Web: Fetch] → content="...architectural details..."

2.3 Vision

The vision node provides on-demand image analysis via the active VLM (Vision Language Model) configured in the LLM profile.

UI Color: Violet

Input formats: - image as numpy array (e.g., from camera, screenshot) - image as base64 string (pre-encoded external images) - prompt as optional override (defaults to "Describe this image concisely for a navigation agent.")

Configuration: - temperature (slider, 0.0–1.0, default 0.3): VLM sampling temperature

Example workflow:

[Episode: Panoramic] → image=<numpy array> → [Vision: Analyze] → response="A hallway with..."
                    → prompt="What objects do you see?"

2.4 Spatial Math

Spatial math nodes compute distances and headings for navigation reasoning.

UI Color: Emerald

Coordinate system: - Habitat uses Y-up convention: horizontal plane is XZ, forward is −Z - Position: [x, y, z] - Orientation: quaternion [qx, qy, qz, qw] (converted to yaw for heading)

Heading interpretation: Relative headings are interpreted as human-readable directions: - < 15°: "ahead" - 15°–60°: "slightly left/right" - 60°–120°: "left/right" - 120°–165°: "behind-left/right" - > 165°: "behind"

Example workflow:

[Episode: State] → position=[1.0, 0.5, 2.0] → [Spatial: Measure Distance]
               → orientation=[0, 0.707, 0, 0.707]      distance="5.12"

[Spatial: Compute Heading] → absolute_deg=45.2, relative_deg=-30.5, interpretation="slightly right"

2.5 Episode Context

Episode context nodes provide access to the Habitat environment's state and history.

UI Color: Rose

Details: - Get Instruction: Queries the HabitatEnvManager (if loaded) via async executor. Returns "(Habitat not loaded)" if environment unavailable. - Get Step Count: Returns ctx.step — the iteration counter in the execution context. - Get History: Maintains a persistent list across steps. Each entry records step number, action name (STOP/FORWARD/LEFT/RIGHT), and a truncated LLM response (120 chars).

Configuration (Get History): - max_history (slider, 5–200, default 50): Maximum history entries to return

Example workflow:

[Trigger] → [Episode: Get Instruction] → instruction="Go to the kitchen."
[LLM] → action=1 → [Episode: Get History] → history="Step 1: FORWARD — Agent moved ahead...\nStep 2: LEFT — Turn to explore..."
     → response="Turning left to see the hallway"

3. Shared Scratch Pad

The scratch pad is implemented as a module-level dictionary (_scratch_pads: dict[str, dict[str, str]]) keyed by pad_id from node configuration.

Isolation and Sharing

• Same pad_id → shared state: All Write/Read/List nodes with pad_id="planning" access the same pad.
• Different pad_id → isolated state: pad_id="planning" and pad_id="memory" are completely separate.
• Default pad_id is "default" if not configured.

Lifecycle

• Initialization: Pads are created on first access via _get_pad(config).
• Clearing: All pads are cleared when the NodeSet is shut down (via BasicAgentNodeSet.shutdown()).

Use this for: - Storing intermediate reasoning steps - Accumulating context across multiple LLM calls - Sharing state between parallel agent branches

Do NOT use for: - Persistent data (cleared on shutdown) - Large data structures (memory overhead)

4. Configuration

Each node exposes configuration fields in the canvas UI. These fields are passed to the node's forward() method via self.config.

Scratch Pad Nodes

{
  "pad_id": "default"
}

Web Grounding Nodes

{
  "max_results": 5,
  "max_chars": 4000
}

Vision Node

{
  "temperature": 0.3
}

Episode Context Nodes

{
  "max_history": 50
}

5. Usage

Loading the NodeSet

Send a POST request to the backend:

POST /api/components/nodesets/basic_agent_tools/load

The backend registers all 11 nodes in the component registry and makes them available in the canvas UI under their respective categories.

Wiring Patterns

Pattern 1: Agent Planning Loop

[Seed] → [Episode: Get Instruction]
      → [Scratch Pad: Write] (key="task", content=instruction)
      → [LLM: Reason]
      → [Scratch Pad: Read] (key="task")
      → [Scratch Pad: List] (to monitor planning state)

Pattern 2: Web-Grounded Navigation

[LLM: Query] → [Web: Search] → [LLM: Analyze] → [Decision]
                                            ↓
                          [Web: Fetch] (fetch top result)
                                            ↓
                          [Vision: Analyze] (if image available)

Pattern 3: Spatial Reasoning

[Episode: State] → [Spatial: Measure Distance] → [LLM: Distance-aware decision]
               → [Spatial: Compute Heading]

Pattern 4: History Tracking

[LLM Output] → [Episode: Get Step Count]
           → [Episode: Get History] (accumulate actions)
           → [LLM: Use history for context]

Error Handling

All nodes include graceful fallbacks: - Empty query → "(empty query)" - Network failure → "(search failed: )" - Habitat not loaded → "(Habitat not loaded)" - Invalid image → "(invalid image input)" - No LLM profile → "(no LLM profile active — cannot call VLM)"

Agents should check for these error patterns in LLM prompts.