Surface Normals NodeSet

The Surface Normals NodeSet (NormalNodeSet, workspace/nodesets/model/model_normal.py) wraps Sapiens normal estimation via transformers AutoModelForNormalEstimation: one RGB frame in, a dense per-pixel unit surface-normal map out. It is the orientation companion to monocular depth — where depth answers "how far", normals answer "which way each surface faces". Ground-plane and wall detection, walkable-surface reasoning, and lifting a scene into oriented 3D structure all want per-pixel normals.

env: ac-fm (shared FM env) · backend: transformers AutoModelForNormalEstimation + AutoImageProcessor · default: facebook/sapiens2-normal-0.4b (ungated)

1. One primitive: RGB → unit normals

The tool takes a list of same-resolution images and returns, per image, a per-pixel unit 3-vector giving the surface orientation in the camera frame. In an agent loop N is typically 1 (per-frame normals); batching is offered for the multi-view case. It mirrors the Depth Anything node exactly — same input contract, same envelope convention — so depth and normals compose as a drop-in geometry pair.

2. Images in — uniform resolution

The images port accepts a list of {rgb_base64} dicts (or raw base64 strings). All images in one call must share resolution — the envelope is a single (N, H, W, 3) buffer, so mixed sizes degrade to empty with a self-log (split into uniform batches). A malformed frame degrades the whole call rather than guessing.

3. Model & post-processing

The engine loads AutoModelForNormalEstimation + AutoImageProcessor for the configured model_id (only Sapiens maps to normal estimation in this transformers version). The raw prediction is bicubic-upsampled back to the original resolution, then each pixel's vector is L2-renormalized to unit length:

normals = model(pixel_values=pixel_values).normals              # (N, 3, h', w')
normals = F.interpolate(normals, size=(H, W), mode="bicubic") # back to original
normals = F.normalize(normals, dim=1).permute(0, 2, 3, 1)   # unit length, (N, H, W, 3)

Each model_id is a lazy singleton in a registry, frozen and moved to the device once; inference is single-flight per engine.

4. Normal envelope

The output is a TEXT JSON envelope — the normal map as a raw C-contiguous float32 buffer, base64-encoded (byte-exact across the server-mode HTTP boundary, ~4× smaller than a JSON float list):

{"shape": [1, 480, 640, 3], "dtype": "float32", "b64": "<buffer>", "model_id": "facebook/sapiens2-normal-0.4b"}

normals[n, y, x] is a unit 3-vector (x, y, z) in the camera frame, at the original input resolution. Every pixel has |n| ≈ 1.

5. Canvas Node

model_normal__estimate_normals

6. Server Mode

parallelism = "shared" (a stateless normal estimator — one server across eval workers); server_python = conda_env_python("ac-fm", "NORMAL_PYTHON"). Normal estimation runs in the shared ac-fm env — AutoModelForNormalEstimation is native to the transformers stack there. The file stays Python-3.8-parseable. Load: POST /api/components/nodesets/model_normal/load?mode=server.

7. Environment Variables

8. Degraded Mode

A weight-load failure latches (_load_failed — no retry storm); that, a missing/malformed frame, or mixed input resolutions yields "" on the node's output. An empty envelope is a clean "no normals this step" signal; consumers keep their own fallback and never receive a fabricated orientation map.