release: v0.2.4 — audit batch 6 (round-0 progress signal, moa config validation, contract/test hardening)

Second-audit functional/contract follow-up. Each code fix is RED-first. Full suite
ALL GREEN (MoA 30, stop-gate 26, driver-limits 17, codex-limits 18). Version bumped
in all three manifest spots.

- fix(drivers): the no-progress fingerprint could not observe round-0 progress
  (round 0 fills PLAN/FEATURE_MATRIX before any runs/round-N.md), so a multi-session
  round 0 false-tripped NO_PROGRESS. Fingerprint now includes round-0 bootstrap
  bytes; identical across both drivers. PL-2.
- fix(moa): config reference_models that is [] (silently aggregator-only) or a
  non-array typo (silent DEFAULT fallback) is now a clean error (exit 1). Absent
  reference_models still uses DEFAULT. MO-2 / MO-3.
- docs(moa-decision): documents the exit-1 contract + full exit-code semantics +
  the capture-stdout-on-output-write-failure rule. PL-3.
- test: covers F6 coordinator-mode env sanitization, the codex driver.log
  writability guard, and LOOP_TESTING_GATE_STALE_SECONDS=0 disabling stale-disarm.

Behavior note: an empty/non-array reference_models now errors instead of silently
degrading — omit it for DEFAULT or give a non-empty array.

Deferred (tracked in roadmap): driver concurrency lock (P3, portable-lock + trap
composition), skill/references install-dir location (P2, LLM-visible metadata needing
a real-loop run).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: sdsrss

.claude-plugin/marketplace.json

@@ -5,14 +5,14 @@
   },
   "metadata": {
     "description": "loop-testing plugin marketplace",
-    "version": "0.2.3"
+    "version": "0.2.4"
   },
   "plugins": [
     {
       "name": "loop-testing",
       "source": "./",
       "description": "Autonomous self-test / self-fix / self-iterate QA loop that drives a finished project to convergence as a simulated real user in a sandbox (自测·验收·QA 循环).",
-      "version": "0.2.3",
+      "version": "0.2.4",
       "category": "development"
     }
   ]

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "loop-testing",
-  "version": "0.2.3",
+  "version": "0.2.4",
   "description": "Autonomous QA self-test loop for a finished project: Claude acts as a real user in a sandbox, exercising every feature to surface bugs, logic/flow/UX/security/hidden issues — fixing the safe low-risk ones with regression tests and atomic commits, and recording the rest with MoA multi-model decision advice — looping self-test → self-fix → self-iterate until two consecutive low-risk rounds converge. Use when the user wants hands-off acceptance testing / 自测 / 自我测试 / 验收 / QA 循环 / self-test loop / 自动测试并修复 of their app (CLI, API, Web, or library). Runs fully autonomous, pausing only for keys/payment/network permission, a suspected security vuln to report, or a total blocker.",
   "author": {
     "name": "sds"

CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## 0.2.4 — 2026-07-12
+
+Audit batch 6 (second-audit follow-up, functional/contract hardening). Lands the
+remaining actionable P2/P3 findings from the second production-readiness audit,
+each RED-first. Full suite `ALL GREEN` (MoA 30, stop-gate 26, driver-limits 17,
+codex-limits 18). Two findings are deferred with rationale (see below).
+
+**Behavior change to note:** an `moa.config.json` whose `reference_models` is an
+empty array or a non-array value now fails with a clean `error:` (exit 1) instead
+of silently running aggregator-only / falling back to the DEFAULT models. If you
+relied on that silent fallback, either omit `reference_models` (to use the DEFAULT
+set) or give it a non-empty array.
+
+- **fix(drivers)**: the no-progress fingerprint (`round | issues | converged_streak
+  | runs count+bytes`) could not observe round-0 progress — round 0 fills PLAN.md +
+  FEATURE_MATRIX.md before any `runs/round-N.md` exists, so a round 0 spanning ≥3
+  sessions on a large target fingerprinted as static and false-tripped NO_PROGRESS
+  (exit 5). The fingerprint now includes round-0 bootstrap bytes (PLAN +
+  FEATURE_MATRIX). Kept identical across both drivers. (+2 driver tests: a round-0
+  bootstrap run reaches --max-sessions instead of NO_PROGRESS.) PL-2.
+- **fix(moa)**: a config `reference_models` that is an empty array (silently ran
+  aggregator-only) or a non-array typo (silently fell back to the paid DEFAULT
+  models) now surfaces as a clean `error:` (exit 1) — aligning with the "zero
+  criteria → refuse" discipline. Absent `reference_models` still uses the DEFAULT.
+  (+2 MoA tests.) MO-2 / MO-3.
+- **docs(moa-decision)**: the reference now documents the exit-1 contract (user-side
+  config/argument/input/output-write errors) alongside 0 and 2, with the full exit-
+  code semantics and the "capture the stdout decision on an --output write failure"
+  rule. PL-3.
+- **test**: closed three previously-uncovered paths — the F6 coordinator-mode env
+  sanitization (the child must not inherit orchestration-only mode vars), the codex
+  driver's driver.log writability guard (unwritable → die exit 2 before any session),
+  and `LOOP_TESTING_GATE_STALE_SECONDS=0` disabling the stale-sentinel auto-disarm.
+
+Deferred (tracked): a driver concurrency lock (needs a portable mkdir-lock + PID
+liveness + trap composition across the two drivers' differing traps; P3), and making
+the skill/references state how to locate the installed script dir (LLM-visible
+metadata requiring a real-loop run to verify; P2).
+
 ## 0.2.3 — 2026-07-12
 
 Second full production-readiness audit (5-track parallel review + per-finding

skills/loop-testing/references/moa-decision.md

@@ -83,11 +83,15 @@ provider 解析：config 可为每个模型单独指定 `provider`；未指定
 
 | 情况 | 引擎行为 | 退出码 |
 |------|---------|--------|
-| 部分参考模型失败（超时 / HTTP 错误 / 响应异常）| 用其余模型继续，元数据记 `partial-references` | 0 |
+| 正常 / 部分参考模型失败（超时 / HTTP 错误 / 响应异常）| 产出决策文件；部分失败时元数据记 `partial-references` | 0 |
 | 全部参考模型失败 | 仅聚合模型出建议，标 `degraded: no-references` | 0 |
 | 聚合模型失败，或根本没有可用 key | 报错到 stderr（已脱敏），**不产出决策文件** | **2** |
+| 用户侧错误：未知 CLI flag、`--config` 不可读 / 非法 JSON / `reference_models` 非非空数组、非法 provider、`--input` 缺失 / 不可读 / 为空、`--output` 写盘失败 | 打干净 `error: <msg>`（**非** `fatal:` 栈），不产出文件；写盘失败时决策已先打到 stdout | **1** |
 
-当引擎 **exit 2** 时（MoA 不可用）：编排 agent 自行用**单模型推理**给出建议，写入 `DEC-NNN.md`，并在文件与元数据中**显式标注 `degraded: single-model`**，说明缺少多模型委员会视角的局限。循环照常继续，绝不因 MoA 不可用而谎报或停摆。
+**退出码语义**：`0` = 成功（含优雅降级）；`1` = 用户侧错误（配置/参数/输入/写盘），修正后重试；`2` = MoA 不可用（付费聚合链断）。
+
+- 引擎 **exit 2** 时（MoA 不可用）：编排 agent 自行用**单模型推理**给出建议，写入 `DEC-NNN.md`，并在文件与元数据中**显式标注 `degraded: single-model`**，说明缺少多模型委员会视角的局限。循环照常继续，绝不因 MoA 不可用而谎报或停摆。
+- 引擎 **exit 1** 时（用户侧错误）：先按 stderr 的 `error:` 修正配置/参数/输入再重试；**若错误是 `--output` 写盘失败，完整决策块已打到 stdout**——把它捕获落盘为 `DEC-NNN.md`，不要丢弃已付费的结论。
 
 真实 key 冒烟属**付费接口 / 外网调用**，按 FR-6.7 需先获授权，不在自主推进范围内。
 

skills/loop-testing/scripts/moa.mjs

@@ -179,7 +179,14 @@ async function resolveConfig(args, env) {
     } catch (e) {
       throw new Error(`config file is not valid JSON: ${configPath}: ${e.message}`);
     }
-    if (Array.isArray(parsed.reference_models)) {
+    // reference_models absent -> keep DEFAULT. Present -> must be a non-empty array:
+    // a non-array typo silently fell back to the DEFAULT (wrong, paid-for) models,
+    // and [] silently ran aggregator-only — both contradict "zero criteria -> refuse"
+    // and are now clean errors instead of silent degradation (audit MO-2 / MO-3).
+    if (parsed.reference_models !== undefined) {
+      if (!Array.isArray(parsed.reference_models) || parsed.reference_models.length === 0) {
+        throw new Error('config "reference_models" must be a non-empty array of model names');
+      }
       cfg.reference_models = parsed.reference_models.map(normalizeModelEntry);
     }
     if (parsed.aggregator != null) cfg.aggregator = normalizeModelEntry(parsed.aggregator);

skills/loop-testing/scripts/unattended-codex.sh

@@ -29,7 +29,8 @@
 #   3  hit --max-sessions before terminal (driver-declared INCOMPLETE).
 #   4  hit --max-minutes before terminal (driver-declared INCOMPLETE).
 #   5  NO_PROGRESS: two consecutive sessions with no change in the composite
-#      progress fingerprint (round | issues | converged_streak | runs count+bytes).
+#      progress fingerprint (round | issues | converged_streak | runs count+bytes |
+#      round-0 bootstrap bytes).
 set -u
 
 PROJECT=""
@@ -92,9 +93,15 @@ runs_sig() { # "<file-count>:<total-bytes>" of runs/*.md — evidence-growth sig
   b=$(cat "$d"/*.md 2>/dev/null | wc -c | tr -d ' ')
   echo "$n:$b"
 }
-progress_sig() { # composite fingerprint: round|issues|streak|runsN:runsB
+bootstrap_sig() { # bytes of round-0 artifacts (PLAN + FEATURE_MATRIX)
+  # Round 0 fills PLAN.md + FEATURE_MATRIX.md BEFORE any runs/round-N.md exists, so
+  # without this a round 0 that spans sessions on a large target fingerprints as
+  # static (round/issues/streak/runs all 0) and false-trips NO_PROGRESS (audit PL-2).
+  cat "$LT/PLAN.md" "$LT/FEATURE_MATRIX.md" 2>/dev/null | wc -c | tr -d ' '
+}
+progress_sig() { # composite fingerprint: round|issues|streak|runsN:runsB|bootstrapB
   local s; s="$(state_field converged_streak)"; [ -n "$s" ] || s=-1
-  printf '%s|%s|%s|%s' "$(round_of)" "$(issue_count)" "$s" "$(runs_sig)"
+  printf '%s|%s|%s|%s|%s' "$(round_of)" "$(issue_count)" "$s" "$(runs_sig)" "$(bootstrap_sig)"
 }
 
 log() { echo "$*" >> "$DLOG"; }
@@ -166,9 +173,10 @@ while true; do
   fi
 
   # No-progress breaker: ANY change in the composite signal (round, issue count,
-  # converged_streak, runs/ evidence bytes+count) counts as progress — catches a
-  # long round spanning sessions and convergence progress the old round+issues
-  # signal misread as stuck (audit A3). Kept identical to unattended-loop.sh.
+  # converged_streak, runs/ evidence bytes+count, round-0 bootstrap bytes) counts as
+  # progress — catches a long round spanning sessions, convergence progress, and a
+  # round 0 filling PLAN/FEATURE_MATRIX before any runs/ file, all of which the old
+  # round+issues signal misread as stuck (audit A3 / PL-2). Identical to unattended-loop.sh.
   if [ "$cur_sig" = "$prev_sig" ]; then
     noprog=$(( noprog + 1 ))
   else
@@ -177,6 +185,6 @@ while true; do
   prev_sig="$cur_sig"
 
   if [ "$noprog" -ge 2 ]; then
-    summary_exit 5 "NO_PROGRESS: two consecutive sessions with no change in round/issues/streak/runs"
+    summary_exit 5 "NO_PROGRESS: two consecutive sessions with no change in round/issues/streak/runs/bootstrap"
   fi
 done

skills/loop-testing/scripts/unattended-loop.sh

@@ -24,7 +24,8 @@
 #   3  hit --max-sessions before terminal (driver-declared INCOMPLETE).
 #   4  hit --max-minutes before terminal (driver-declared INCOMPLETE).
 #   5  NO_PROGRESS: two consecutive sessions with no change in the composite
-#      progress fingerprint (round | issues | converged_streak | runs count+bytes).
+#      progress fingerprint (round | issues | converged_streak | runs count+bytes |
+#      round-0 bootstrap bytes).
 set -u
 
 PROJECT=""
@@ -90,10 +91,16 @@ runs_sig() { # "<file-count>:<total-bytes>" of runs/*.md — evidence-growth sig
   b=$(cat "$d"/*.md 2>/dev/null | wc -c | tr -d ' ')
   echo "$n:$b"
 }
-progress_sig() { # composite progress fingerprint: round|issues|streak|runsN:runsB
+bootstrap_sig() { # bytes of round-0 artifacts (PLAN + FEATURE_MATRIX)
+  # Round 0 fills PLAN.md + FEATURE_MATRIX.md BEFORE any runs/round-N.md exists, so
+  # without this a round 0 that spans sessions on a large target fingerprints as
+  # static (round/issues/streak/runs all 0) and false-trips NO_PROGRESS (audit PL-2).
+  cat "$LT/PLAN.md" "$LT/FEATURE_MATRIX.md" 2>/dev/null | wc -c | tr -d ' '
+}
+progress_sig() { # composite fingerprint: round|issues|streak|runsN:runsB|bootstrapB
   local s
   s="$(state_field converged_streak)"; [ -n "$s" ] || s=-1
-  printf '%s|%s|%s|%s' "$(round_of)" "$(issue_count)" "$s" "$(runs_sig)"
+  printf '%s|%s|%s|%s|%s' "$(round_of)" "$(issue_count)" "$s" "$(runs_sig)" "$(bootstrap_sig)"
 }
 
 mkdir -p "$LT"
@@ -182,17 +189,18 @@ while true; do
   fi
 
   # 4. No-progress circuit breaker. Progress = ANY change in the composite signal
-  #    (round, issue count, converged_streak, or runs/ evidence bytes+count). The
-  #    last two catch a long round that spans sessions appending evidence before
-  #    `round` ticks, and progress made by advancing convergence — cases the old
-  #    round+issues-only signal misread as stuck (audit A3).
+  #    (round, issue count, converged_streak, runs/ evidence bytes+count, or round-0
+  #    bootstrap bytes). These catch a long round that spans sessions appending
+  #    evidence before `round` ticks, progress made by advancing convergence, and a
+  #    round 0 filling PLAN/FEATURE_MATRIX before any runs/ file exists — cases the
+  #    old round+issues-only signal misread as stuck (audit A3 / PL-2).
   if [ "$cur_sig" = "$prev_sig" ]; then
     no_progress=$(( no_progress + 1 ))
   else
     no_progress=0
   fi
   prev_sig="$cur_sig"
   if [ "$no_progress" -ge 2 ]; then
-    summary_exit 5 "NO_PROGRESS: 2 consecutive sessions with no change in round/issues/streak/runs (stuck)"
+    summary_exit 5 "NO_PROGRESS: 2 consecutive sessions with no change in round/issues/streak/runs/bootstrap (stuck)"
   fi
 done

tests/driver/codex-lib.sh

@@ -48,7 +48,15 @@ round=-1
 [ -f "$STATE" ] && round=$(grep -aE '^round:' "$STATE" | head -1 | sed 's/[^0-9-]//g')
 case "$round" in ''|*[!0-9-]*) round=-1 ;; esac
 streak=0
-if [ "${STUB_STREAK_ONLY:-0}" = "1" ]; then
+if [ "${STUB_BOOTSTRAP:-0}" = "1" ]; then
+  # Round-0 progress: round / issue count / converged_streak stay static and NO
+  # runs/round-N.md is created, but PLAN.md + FEATURE_MATRIX.md grow each call
+  # (the project-analysis phase before the loop proper). Exercises PL-2.
+  new_round=$round; [ "$new_round" -lt 0 ] && new_round=0
+  printf 'plan detail line for session\n' >> "$LT/PLAN.md"
+  printf '| feature | entry | scenario | PASS | R0 |\n' >> "$LT/FEATURE_MATRIX.md"
+  status=RUNNING
+elif [ "${STUB_STREAK_ONLY:-0}" = "1" ]; then
   # Progress via convergence + evidence only: round and issue count stay static,
   # but converged_streak advances and a runs/ evidence file grows each call.
   new_round=$round; [ "$new_round" -lt 0 ] && new_round=0

tests/driver/codex-limits.test.sh

@@ -78,4 +78,22 @@ STUB_NO_STATE=1 bash "$CODEX_DRIVER" --project "$WS7" --codex-bin "$stub" --no-p
 assert_rc $? 5 "absent STATE.md -> exit 5"
 assert_eq "1" "$(sessions_in_log "$WS7")" "exits after exactly 1 STATE-less session (not 2)"
 
+# J. Round-0 progress: round/issues/streak static and NO runs/ file, but PLAN.md +
+#    FEATURE_MATRIX.md grow each session. Must NOT trip NO_PROGRESS (audit PL-2).
+#    Mirrors loop driver H.
+WS8=$(mk_proj); trap 'rm -rf "$WS" "$WS2" "$WS3" "$WS4" "$WS5" "$WS6" "$WS7" "$WS8"' EXIT
+stub=$(write_stub "$WS8")
+write_state "$WS8" RUNNING 0
+STUB_BOOTSTRAP=1 bash "$CODEX_DRIVER" --project "$WS8" --codex-bin "$stub" --no-protect --max-sessions 3 >/dev/null 2>&1
+assert_rc $? 3 "round-0 bootstrap progress (PLAN/FEATURE_MATRIX grow) -> max-sessions, not NO_PROGRESS"
+
+# K. driver.log not writable -> die exit 2 BEFORE any session (parity with the loop
+#    driver's writability guard; R16 gained this for codex but had no dedicated test).
+WS9=$(mk_proj); trap 'rm -rf "$WS" "$WS2" "$WS3" "$WS4" "$WS5" "$WS6" "$WS7" "$WS8" "$WS9"' EXIT
+mkdir "$WS9/docs/looptesting/driver.log"   # append to a directory fails -> guard fires
+stub=$(write_stub "$WS9")
+bash "$CODEX_DRIVER" --project "$WS9" --codex-bin "$stub" --no-protect --max-sessions 1 >/dev/null 2>&1
+assert_rc $? 2 "unwritable driver.log -> die exit 2"
+assert_eq "0" "$(sessions_in_log "$WS9")" "no session launched when driver.log is unwritable"
+
 report "codex-limits.test.sh"

tests/driver/driver-limits.test.sh

@@ -58,4 +58,39 @@ STUB_NO_STATE=1 bash "$DRIVER" --project "$WS5" --claude-bin "$stub" --max-sessi
 assert_rc $? 5 "absent STATE.md -> exit 5"
 assert_eq "1" "$(sessions_in_log "$WS5")" "exits after exactly 1 STATE-less session (not 2)"
 
+# H. Round-0 progress: round/issues/streak static and NO runs/ file, but PLAN.md +
+#    FEATURE_MATRIX.md grow each session (project-analysis phase). Must NOT trip
+#    NO_PROGRESS — the round+issues+streak+runs signal alone misread a multi-session
+#    round 0 as stuck (audit PL-2).
+WS6=$(mk_proj); trap 'rm -rf "$WS" "$WS2" "$WS3" "$WS4" "$WS5" "$WS6"' EXIT
+stub=$(write_stub "$WS6")
+write_state "$WS6" RUNNING 0
+STUB_BOOTSTRAP=1 bash "$DRIVER" --project "$WS6" --claude-bin "$stub" --max-sessions 3 >/dev/null 2>&1
+assert_rc $? 3 "round-0 bootstrap progress (PLAN/FEATURE_MATRIX grow) -> runs to max-sessions, not NO_PROGRESS"
+
+# I. F6 coordinator-mode env sanitization: vars that would boot the child session in
+#    orchestration-only mode (Agent/SendMessage/TaskStop/Workflow, no Read/Bash/Edit/
+#    Write) must NOT reach the child. The stub records which survived into its own env.
+WS7=$(mk_proj); trap 'rm -rf "$WS" "$WS2" "$WS3" "$WS4" "$WS5" "$WS6" "$WS7"' EXIT
+cat > "$WS7/envstub.sh" <<'ESTUB'
+#!/usr/bin/env bash
+set -u
+env | grep -aE '^CLAUDE_CODE_(COORDINATOR_MODE|EXPERIMENTAL_AGENT_TEAMS|CHILD_SESSION|SESSION_ID)=' \
+  > docs/looptesting/child-env.txt || true
+cat > docs/looptesting/STATE.md <<EOS
+# STATE
+round: 0
+converged_streak: 2
+status: CONVERGED
+max_rounds: 12
+EOS
+exit 0
+ESTUB
+chmod +x "$WS7/envstub.sh"
+CLAUDE_CODE_COORDINATOR_MODE=1 CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 \
+CLAUDE_CODE_CHILD_SESSION=1 CLAUDE_CODE_SESSION_ID=sess-abc \
+  bash "$DRIVER" --project "$WS7" --claude-bin "$WS7/envstub.sh" --max-sessions 1 >/dev/null 2>&1
+assert_rc $? 0 "converged after a sanitized session -> exit 0"
+assert_eq "" "$(cat "$WS7/docs/looptesting/child-env.txt" 2>/dev/null)" "coordinator-mode env vars unset for the child (F6)"
+
 report "driver-limits.test.sh"