ADAS on AgentCanvas — paper analysis + adas-subagent port

A two-part document. Section 1 dissects Hu et al. 2024 "Automated Design of Agentic Systems" — how search.py's outer loop, archive injection, and debug retry actually work, with the original source annotated. Section 2 documents the AgentCanvas port (adas-subagent): one main Claude conversation envelope plus one Agent sub-agent per Reflexion round, giving 3 independent tool-augmented samples per propose.

section 1 upstream paper analysis   section 2 adas-subagent (reasoning module = sub-agent)   scope coding-agent-era extension, no AAS additions

Framing note. Upstream ADAS lives in the reasoning module = stateless LLM call paradigm — the only option in 2024. adas-subagent proposes reasoning module = tool-augmented Claude conversation with context management as the coding-agent-era replacement. This is the primary contribution; "faithful 1:1 reproduction" is not the goal. Several paradigm-independent elements (3-call Reflexion structure, Reflexion_prompt_1 / 2 text, bootstrap_CI algorithm, archive contract) are preserved verbatim — the "verbatim" labels on individual tables below are accurate at that per-element level. Any residual “faithful port”-style language elsewhere should be read as “structurally accurate translation of paradigm-independent elements,” not as a 1:1 fidelity claim.

Section 1 — Upstream paper analysis (Hu et al. 2024)

How the original ADAS paper's search() function drives one generation (source: third_party/ADAS/_mmlu/search.py:146–242). The orange 3-call Reflexion sequence is the load-bearing piece — our adas-subagent proposer skill (Section 2) preserves it verbatim — three independent Agent sub-agent spawns, one per Reflexion round. Annotations below the SVG link each diagram region to the exact upstream line.

Annotated source — does the diagram match?

Below is the verbatim search() function from third_party/ADAS/_mmlu/search.py:146–241, with right-side annotations tying each region back to a box / arrow in the diagram above. Yes — the diagram is a faithful structural translation; every diagram element has a corresponding code region marked here.

Diagram ↔ code map:

• Top row (3 boxes) ↔ 156, 159–172, 178 — init archive + seed scoring + persist.
• Outer dashed for-loop box ↔ 180.
• 3 orange LLM boxes ↔ 188, 194, 198 — each is one get_json_response_from_gpt_reflect sharing the same msg_list.
• Red "any LLM raises" skip annotation ↔ 199–203.
• Inner orange dashed debug box ↔ 206.
• evaluate_forward_fn box + decision diamond ↔ 208–212.
• YES branch (retry feedback arrow) ↔ 215–218 — push err to msg_list, re-ask LLM.
• "3 retries exhausted" red skip box ↔ 224–226.
• NO branch → bootstrap_CI → archive.append ↔ 228 and 236.

146  def search(args):
147      file_path = os.path.join(args.save_dir, f"{args.expr_name}_run_archive.json")
148      if os.path.exists(file_path):
149          with open(file_path, 'r') as json_file:
150              archive = json.load(json_file)                       # resume from disk if exists
151          if "generation" in archive[-1] and isinstance(archive[-1]['generation'], int):
152              start = archive[-1]['generation']
153          else:
154              start = 0
155      else:
156          archive = get_init_archive()                             ◀── BOX: get_init_archive (7 baseline agents)
157          start = 0
158
159      for solution in archive:                                     ◀── BOX: evaluate + bootstrap_CI (score each seed)
160          if 'fitness' in solution:
161              continue
162
163          solution['generation'] = "initial"
164          print(f"============Initial Archive: {solution['name']}=================")
165          try:
166              acc_list = evaluate_forward_fn(args, solution["code"])
167          except Exception as e:
168              print("During evaluating initial archive:")
169              print(e)
170              continue
171
172          fitness_str = bootstrap_confidence_interval(acc_list)
173          solution['fitness'] = fitness_str
174
175          # save results
176          os.makedirs(os.path.dirname(file_path), exist_ok=True)
177          with open(file_path, 'w') as json_file:
178              json.dump(archive, json_file, indent=4)              ◀── BOX: archive.json (seeded archive persisted)
179
180      for n in range(start, args.n_generation):                    ◀── OUTER DASHED BOX (for n in range(n_generation))
181          print(f"============Generation {n + 1}=================")
182          system_prompt, prompt = get_prompt(archive)              ◀── BOX: get_prompt(archive)
183          msg_list = [
184              {"role": "system", "content": system_prompt},
185              {"role": "user", "content": prompt},
186          ]
187          try:
188              next_solution = get_json_response_from_gpt_reflect(msg_list, args.model)
                                                                     ◀── BOX (orange): LLM #1 propose
                                                                         temp=0.8 / JSON mode / max_tokens=4096 (defaults in search.py:53–67)
189
190              Reflexion_prompt_1, Reflexion_prompt_2 = get_reflexion_prompt(archive[-1] if n > 0 else None)
191              # Reflexion 1
192              msg_list.append({"role": "assistant", "content": str(next_solution)})
193              msg_list.append({"role": "user", "content": Reflexion_prompt_1})
194              next_solution = get_json_response_from_gpt_reflect(msg_list, args.model)
                                                                     ◀── BOX (orange): LLM #2 Reflexion_prompt_1
195              # Reflexion 2
196              msg_list.append({"role": "assistant", "content": str(next_solution)})
197              msg_list.append({"role": "user", "content": Reflexion_prompt_2})
198              next_solution = get_json_response_from_gpt_reflect(msg_list, args.model)
                                                                     ◀── BOX (orange): LLM #3 Reflexion_prompt_2
                                                                         next_solution = {thought, name, code}
199          except Exception as e:
200              print("During LLM generate new solution:")
201              print(e)
202              n -= 1                                               ◀── RED SKIP: any of LLM #1/#2/#3 raised
203              continue                                                           n -= 1; continue (generation NOT archived)
204
205          acc_list = []
206          for _ in range(args.debug_max):                          ◀── INNER ORANGE DASHED BOX (debug_max=3)
207              try:
208                  acc_list = evaluate_forward_fn(args, next_solution["code"])
                                                                     ◀── BOX: evaluate_forward_fn
209                  if np.mean(acc_list) < 0.01 and SEARCHING_MODE:  ◀── DIAMOND: mean(acc)<0.01? → "yes" path
210                      raise Exception("All 0 accuracy")
211                  break                                            ◀── DIAMOND: no → leave debug loop, success
212              except Exception as e:                               ◀── DIAMOND: yes → catch & retry
213                  print("During evaluation:")
214                  print(e)
215                  msg_list.append({"role": "assistant", "content": str(next_solution)})
216                  msg_list.append({"role": "user", "content":      ◀── YES branch arrow: push err to msg_list
                      f"Error during evaluation:\n{e}\nCarefully consider where you went wrong in your latest implementation. "
                      f"Using insights from previous attempts, try to debug the current code to implement the same thought. "
                      f"Repeat your previous thought in 'thought', and put your thinking for debugging in 'debug_thought'"})
217                  try:                                             ◀── ask LLM (with debug_thought field) → new code
218                      next_solution = get_json_response_from_gpt_reflect(msg_list, args.model)
219                  except Exception as e:
220                      print("During LLM generate new solution:")
221                      print(e)
222                      continue
223                  continue
224          if not acc_list:                                         ◀── RED SKIP: 3 retries exhausted
225              n -= 1                                                             n -= 1
226              continue                                                           continue (generation NOT archived)
227
228          fitness_str = bootstrap_confidence_interval(acc_list)    ◀── BOX: bootstrap_CI → fitness_str
229          next_solution['fitness'] = fitness_str
230          next_solution['generation'] = n + 1
231
232          if 'debug_thought' in next_solution:
233              del next_solution['debug_thought']                   # clean scratch field before archiving
234          if 'reflection' in next_solution:
235              del next_solution['reflection']
236          archive.append(next_solution)                            ◀── BOX: archive.append(next_solution)
237
238          # save results
239          os.makedirs(os.path.dirname(file_path), exist_ok=True)
240          with open(file_path, 'w') as json_file:
241              json.dump(archive, json_file, indent=4)              # archive grows iff success

Upstream invariants worth marking — these are the paradigm-independent structural anchors adas-subagent preserves verbatim, even as the reasoning module mechanism itself is upgraded:

• 3 sequential LLM calls share one msg_list — R1 sees the assistant's propose response; R2 sees both. adas-subagent's proposer § 6 preserves this verbatim across the three Agent sub-agent spawns.
• Failed generations are NOT appended to archive — LLM exceptions and exhausted retries both n -= 1; continue. adas-subagent commits only via the loop skill's Atomic Writer, which fires only on full success; SKIP paths do rm -rf .staging/iter_n/ with no archive append. Section 2 documents this contract.
• Fitness is a bootstrap-95%-CI string, e.g. "95% Bootstrap CI: (52.3%, 73.1%), Median: 63.0%". adas-subagent's loop computes this verbatim in the Atomic Writer enrichment step; the string is stored at the top of metrics.json AND injected into every archive.jsonl entry so meta-LLM sees uncertainty, not just point estimates.
• Initial archive contains 7 hand-designed agents (COT, COT_SC, Reflexion, LLM_debate, Take_a_step_back, QD, Role_Assignment), each scored before generation 0 starts. adas-subagent keeps the 7-pattern injection but as text-only seed entries (fitness: null) — pre-evaluating 7 graph baselines at 100 ep each would cost ~14 h, infeasible at our scale. Meta-LLM sees the patterns as "design palette", not benchmarked alternatives.
• mean(acc_list) < 0.01 is an upstream debug trigger, same as a Python crash. adas-subagent's implementer intentionally drops this trigger — low / zero metric values on a clean run are real data ("this approach scored 0%"), not a broken-code signal. See Section 2 bucket D (intentional divergences) for rationale.

Section 2 below documents how adas-subagent implements each of these invariants concretely.

Section 2 — adas-subagent (reasoning module = sub-agent)

.claude/commands/architect/adas-subagent/ is the AgentCanvas port of upstream ADAS. Structural delta vs upstream: each meta-LLM call from search.py (the 3 Reflexion rounds in proposer and the retry-on-failure call in implementer) becomes one Agent({subagent_type:"general-purpose", ...}) invocation — a fully tool-augmented independent Claude conversation, not a stateless OpenAI API call. The 5-skill layout, archive contract, two-tier evaluation, atomic writer, and vN/iter_M run-dir are kept; bootstrap_CI + fitness_str + Reflexion prompt text are byte-for-byte verbatim.

This section documents:

• The main per-iter flow — understand → loop → proposer (3 Agent spawns) → implementer (native edit + smoke + runtime-fail retry) → evaluator → atomic writer
• The sub-agent contract: independent sampling + full tools + JSON-only deliverable
• Verbatim Reflexion_prompt_1 / 2 text and the cross-spawn msg_list rendering
• What's preserved verbatim from upstream, what's forced by the new substrate, and what's an intentional design choice (the 4-bucket delta table)

The diagram below is the main per-iter flow. The proposer panel contains three Agent sub-agent rectangles (orange-bordered) for R0 / R1 / R2; the implementer panel is method-free, with one fresh retry sub-agent spawned per smoke runtime fail (no mean<0.01 trigger, no msg_list inheritance from proposer, retry_max is a config knob).

At-a-glance — cycle + working memory

Same shape as the myloop reference diagram: per-iter phases run top-to-bottom on the left, the persistent archive lives on the right, dashed lines show what each phase reads from or writes to it. The large flow diagram immediately below preserves the per-skill mechanics (R0/R1/R2 sub-agent spawns inside proposer, smoke retry inside implementer, Atomic Writer); this view is the 10-second summary.

Detailed per-iter flow

Sub-agent contract

Every Agent(...) spawn in adas-subagent obeys three rules:

1. Independent sample. Each spawn is a fresh Claude with its own sampling. R0, R1, R2 are three independent samples, not one autoregressive trace. This is the property a single Claude pass cannot provide and the reason Agent spawns exist at all.
2. Full tool access. subagent_type: "general-purpose", so the sub-agent has Read / Grep / Bash / WebSearch / nested Agent. It can ground its proposal in log.jsonl, archive entries, source under workspace/nodesets/, related literature, etc. This is what the "Claude conversation as modernized meta-LLM" framing delivers — without tool access, sub-agents collapse back to stateless LLM calls.
3. JSON-only deliverable. Each sub-agent MUST emit a single fenced ```json block as the final visible chunk of its return message. Proposer's R0 / R1 / R2 spawns share the same msg_list — each parsed response is appended as {"role":"assistant", "content":json.dumps(parsed)} and fed to the next Agent spawn as text. Implementer's retry spawns do NOT inherit this msg_list; each retry is a fresh coding-agent debugging task that sees the proposal + accumulated failure trace as read-only text, edits the active_workspace/ overlay natively with Edit / Write, and emits {edit_summary, extra_targets} (no Reflexion-style debug_thought field). Proposer's sub-agents MUST NOT edit files — they only produce the change spec; the implementer sub-agent is the one that edits, and only ever under active_workspace/, never frozen workspace/*.

Verbatim Reflexion prompts

Reflexion_prompt_1 and Reflexion_prompt_2 are copied character-for-character from upstream third_party/ADAS/_mmlu/mmlu_prompt.py:497–528 into adas-subagent/proposer.md § 6. The only character-level adaptation is "code" → "patch" in the response-schema description (our artifact is a {intent, targets} change spec, not a Python function string).

The [EXAMPLE] placeholder in Reflexion_prompt_1 follows upstream's get_reflexion_prompt formula verbatim: filled with "Here is the previous agent you tried:\n" + json.dumps(archive[-1]) + "\n\n" for generation ≥ 1, empty string for iter_1 (when there is no prior evolved generation).

msg_list across sub-agents

Upstream search.py threads one Python list of {role, content} dicts through three sequential API calls. adas-subagent keeps the textual content of this list identical, with one unavoidable adaptation: the Agent tool accepts only a single user-string prompt, so prior role: assistant outputs are serialized as text inside subsequent sub-agents' prompts (rendered with [role] block markers). From the sub-agent's perspective this is the same information; the only loss is the literal role: assistant tagging, which the cumulative-prompt-text form approximates.

=== msg_list so far (each [role] block is one conversation turn) ===

[system]
You are a helpful assistant. Make sure to return in a WELL-FORMED JSON object.

[user]
{base prompt: # Overview ... # Task ... # The utility code ... # Discovered architecture archive [ARCHIVE] ... # Output Instruction ...}

[assistant]   ⇐ R0's response, present from R1 onward
{"thought": "...", "name": "...", "patch": {...}}

[user]   ⇐ Reflexion_prompt_1 (verbatim), present from R1 onward
"[EXAMPLE]Carefully review the proposed new architecture and reflect on the following points: ..."

[assistant]   ⇐ R1's response, present from R2 onward
{"reflection": "...", "thought": "...", "name": "...", "patch": {...}}

[user]   ⇐ Reflexion_prompt_2 (verbatim), present at R2 only
"Using the tips in '## WRONG Implementation examples' section, revise the patch further..."

=== end msg_list ===

adas-subagent vs upstream ADAS — the full delta

Against the original ADAS paper (Hu et al. 2024, MMLU codebase): what do we keep verbatim, what's forced by the new substrate, and what's an intentional design choice. Four buckets:

A. Preserved verbatim from upstream

These items survive the port unchanged at the structural or byte-for-byte level. Reasoning module no longer being a stateless API call doesn't disturb them.

B. Adapted by paradigm shift (stateless API → tool-augmented Claude)

These items change because the meta-LLM is no longer a stateless OpenAI API call. The reasoning module is now a tool-augmented Claude sub-agent (Agent({subagent_type: "general-purpose", ...})); the upstream knobs that control stateless sampling don't have a meaningful home in this paradigm.

C. Adapted by AgentCanvas substrate + VLN evaluation cost

These items change because the target environment (Habitat VLN episodes) is fundamentally different from upstream's target (MMLU multiple-choice). VLN episodes are expensive (~1–2 h for 100 ep) and the artifact is a multi-file graph + nodeset spec, not a Python function string.

D. Intentional divergences from upstream (2026-05 refactor)

These changes were not forced by substrate or cost — they're judgement calls about what coding-agent capability changes mean for the search loop. Modern Claude sub-agents are strong enough debuggers that the Reflexion-style retry chain in upstream's debug loop offered little marginal value over fresh-spawn-with-failure-trace. The refactor below removes those ADAS-fidelity-but-not-quality elements from the implementer; the proposer's Reflexion chain (bucket A) is preserved because that's where the quality-diversity actually lives.