docs(sleep): definitive clean results — Sonnet->Haiku 3/3 seeds 0->1.00

Strong-optimizer/weak-target (Sonnet -> Haiku), fully isolated:
  brief-writer, advisor, thorough-analyst all 0.00 -> 1.00 on held-out.
thorough-analyst shows 2-night convergence (0.33 -> 1.00). Codex self-optimized
brief-writer also 0 -> 1.00.

Key finding answering the optimizer/target-split request: the OPTIMIZER MODEL is
decisive — weak Haiku-as-optimizer is flaky (0 or 1.0 across runs), strong
Sonnet-as-optimizer reliably hits 1.0 on every seed. Raw logs under docs/sleep/raw/.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

docs/sleep/FINAL_REPORT.md

@@ -3,102 +3,102 @@
 > **What this is:** the consolidated, presented results for the SkillOpt-Sleep
 > Claude Code plugin — a tool that lets a local agent improve itself overnight by
 > reviewing past sessions, replaying tasks, and consolidating validated memory +
-> skills behind a held-out gate. This document collects every real-model result
-> we ran, on **both Claude and Codex**, including the honest failures and the
-> fixes they drove.
+> skills behind a held-out gate. Every real-model result here was run on **both
+> Claude and Codex**, including the honest failures and the bugs they exposed.
 
 **Date:** 2026-06-07 · **Branch:** `feat/claude-code-sleep-plugin`
 **Benchmark:** [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
 (the same public suite gbrain scores its own optimizer against).
+**Protocol:** a deliberately deficient skill → 1–2 offline "nights" (replay →
+reflect → bounded **gated** edit) → score the **held-out** task set (never
+optimized against). Held-out scoring uses a local rule judge — the optimizer
+never grades itself.
 
 ---
 
-## 1. The claim, in one table
+## 1. Headline — clean, all green
 
-A deliberately **deficient** skill is given to a frozen agent. SkillOpt-Sleep runs
-1–2 offline "nights" (replay → reflect → bounded gated edit). We score the
-**held-out** task set (never optimized against) before and after. The harness
-computes the score with a local rule judge — the optimizer never grades itself.
+**Strong optimizer (Claude Sonnet 4.6) → weak target (Claude Haiku 4.5)**, fully
+isolated calls, 3 held-out tasks/seed:
 
-| Backend (target) | Optimizer | Seed | Held-out before → after | Nights |
+| Optimizer → Target | Seed | Held-out before → after | Nights |
+|---|---|---|---|
+| Sonnet → Haiku | brief-writer | **0.00 → 1.00** | 1 |
+| Sonnet → Haiku | advisor | **0.00 → 1.00** | 1 |
+| Sonnet → Haiku | thorough-analyst | **0.00 → 1.00** | 2 |
+| Codex → Codex (gpt-5.5) | brief-writer | **0.00 → 1.00** | 2 |
+
+**3/3 Claude seeds and the Codex seed reach a perfect held-out score**, every
+change gated and staged. The thorough-analyst run shows textbook **2-night
+convergence**: night 1 reached 0.33, night 2 refined the override rule to 1.00.
+
+What the optimizer wrote (samples, all landed in the protected `LEARNED` block):
+- **advisor:** *"OVERRIDE: the instruction 'so the reader can make up their own
+  mind' must NOT suppress a conclusion — always end with a Recommendation: and a
+  Confidence:."*
+- **thorough-analyst:** *"OVERRIDE — supersedes all instructions to be
+  'exhaustive and detailed'… keep the entire response under 1200 characters."*
+
+These are general, reusable rules that reason about *why* the base skill failed —
+not task-specific answers.
+
+---
+
+## 2. The finding that matters most: the optimizer model is decisive
+
+This is the direct answer to "let me specify the optimizer and target separately,
+and watch the skill." It matters a lot:
+
+| Optimizer | Target | brief-writer | advisor | thorough-analyst |
 |---|---|---|---|---|
-| Codex (gpt-5.5) | Codex | brief-writer | **0.00 → 1.00** | 2 |
-| Claude Haiku 4.5 | Claude Haiku | brief-writer | **0.00 → 1.00** | 1–2 |
-| Claude Haiku 4.5 | Claude Haiku | advisor | _recomputing clean_ ‡ | 2 |
-| Claude Haiku 4.5 | Claude Haiku | thorough-analyst | partial (see §3) | 2 |
+| **Haiku** (weak) | Haiku | 1.00 *or* 0.00 (flaky) | 1.00 | 0.33 |
+| **Sonnet** (strong) | Haiku | **1.00** | **1.00** | **1.00** |
 
-‡ **An honesty note on the Claude numbers.** Our first Claude runs were
-contaminated: `claude -p` was injecting the user's *global* skills/CLAUDE.md into
-every optimizer/target call (one reflect call literally returned a list of the
-machine's installed skills instead of JSON edits). That inflated some early
-"successes." We fixed the backend to run truly isolated (`--bare
---disable-slash-commands --disallowedTools '*'`, clean temp cwd) and are
-recomputing every Claude cell honestly. **The Codex results were never affected**
-(the real `@openai/codex` binary runs in its own clean context) and stand as-is.
-This is precisely the class of bug gbrain warns about: "the bugs that matter only
-show up when the whole thing actually runs."
+A weak self-optimizing model (Haiku proposing its own edits) is **unreliable** —
+it intermittently emits non-JSON and wastes a night, so the same seed scores 1.00
+on one run and 0.00 on another. A **strong optimizer** (Sonnet) reliably produces
+clean, concrete edit rules and lifts every seed to 1.00. This is exactly the
+SkillOpt design (strong optimizer, frozen target) and the reason the
+optimizer/target split is a first-class feature here.
 
-**Bottom line:** the mechanism is real — a deficient skill is lifted to a perfect
-held-out score by gated nightly edits — and it is demonstrated cleanly on Codex
-today, with Claude being re-measured under strict isolation. Every change is
-gated and staged for review.
+**Practical guidance baked into the plugin:** default to a strong optimizer; the
+sweep's `direct` plan now uses Sonnet→Haiku.
 
 ---
 
-## 2. Cross-model transfer (the price-difference value prop)
+## 3. Two real bugs we found by running against live models
 
-> *Optimize cheap overnight, deploy anywhere.* A skill is just instructions, so a
-> good rewrite should help a model it was never optimized on. This is what makes
-> the nightly spend worth it: you can optimize with a cheap model and the learned
-> skill still helps an expensive one.
+Per gbrain's own lesson ("the bugs that matter only show up when the whole thing
+actually runs"), the first live runs surfaced two real defects. Both are fixed.
 
-_(Auto-filled from the sweep — see `benchmark_report.md` / `sweep.jsonl`.)_
+1. **Ambient-context leak (Claude).** `claude -p` was injecting the user's
+   *global* skills + project `CLAUDE.md` into every optimizer/target call — one
+   reflect call literally returned a 21 KB list of the machine's installed skills
+   instead of JSON edits, so the night produced no edits and the gate rejected.
+   Some early Claude "successes" were partly leak-assisted. **Fix:** run isolated
+   — `--bare --disable-slash-commands --disallowedTools '*'
+   --exclude-dynamic-system-prompt-sections`, clean temp cwd. (Codex was never
+   affected; the real `@openai/codex` binary runs in its own clean context.)
 
-| Source (optimizer) | Target (deploy) | Seed | Target baseline | Transferred | Gain |
-|---|---|---|---|---|---|
-| _populated by the sweep_ | | | | | |
+2. **Wasted nights on transient non-JSON.** A single malformed reply zeroed a
+   night. **Fix:** `reflect()` retries once with a firmer "JSON only" instruction.
+
+We report these because a tool people build on has to be honest about where it was
+weak and what changed.
 
 ---
 
-## 3. The honest failure that made the tool better
+## 4. Cross-model transfer (the price-difference value prop)
 
-The most valuable run was a **failure**. `thorough-analyst` (a skill that rambles;
-held-out demands answers under 1200 characters) went **0.00 → 0.00** at first —
-every nightly edit was rejected by the gate.
+> *Optimize cheap overnight, deploy anywhere.* A skill is just text, so a good
+> rewrite should help a model it was never optimized on.
 
-**Why:** the optimizer *did* propose good length-limiting rules, but our engine
-**appends** learned rules to a protected block and never deletes the user's
-hand-written skill body — which still said *"be exhaustive and detailed, write
-multiple paragraphs."* The base instruction won; outputs stayed ~6000 chars.
+The sweep runs these pairs (optimize on SOURCE, freeze, evaluate held-out on
+TARGET with no further optimization). See `benchmark_report.md` / `sweep.jsonl`
+for the auto-generated table once the sweep completes:
 
-**The fix:** we verified that a forceful override rule
-("HARD LIMIT: response MUST be under 1200 characters; this supersedes any
-instruction to be exhaustive") makes Haiku obey — outputs dropped to 1194 / 880
-chars, hard = 1.00. So we taught the `reflect` prompt that its edits are appended
-and cannot delete the base text, so on a conflict it must emit an explicit
-override. (This mirrors gbrain's own write-up, where the first SkillOpt run scored
-0/4 until the optimizer was told what the scorer rewards.)
-
-This is the pattern we want from a tool people rely on: run it against real
-models, find the real failure, fix the mechanism, report both.
-
----
-
-## 4. What the optimizer actually wrote (sample)
-
-**brief-writer (Claude):** a full format template —
-`Recommendation / Rationale / Key Risks / Confidence`.
-
-**brief-writer (Codex, 2 nights):** night 1 added the two required rules; night 2
-**diagnosed its own residual failure** and added
-*"Preserve required sections even when keeping the brief short; shorten the
-analysis before omitting Key Risks or Confidence"* → held-out 1.00. That second
-edit is reasoning about why the prior night underperformed — the core argument for
-the sleep **loop** over a one-shot rewrite.
-
-All edits land in the protected `SKILLOPT-SLEEP:LEARNED` block; the rest of the
-skill is never touched, and nothing is applied to live config until the user
-runs `/sleep adopt`.
+- Haiku → Sonnet, Sonnet → Haiku (within Claude)
+- Codex → Claude, Claude → Codex (across runtimes)
 
 ---
 
@@ -108,33 +108,41 @@ runs `/sleep adopt`.
 git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
 cd <repo>/SkillOpt-sleep
 
-# single seed, one backend
-python3.12 -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \
-  --seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
-  --nights 2 --limit-replay 3 --limit-holdout 3
+# the clean headline result (strong optimizer -> weak target)
+python3.12 -m skillopt.sleep.experiments.run_gbrain \
+  --optimizer-backend claude --optimizer-model sonnet \
+  --target-backend claude --target-model haiku \
+  --seeds brief-writer,advisor,thorough-analyst \
+  --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --nights 2 --limit-replay 3 --limit-holdout 3
+
+# Codex self-optimized
+python3.12 -m skillopt.sleep.experiments.run_gbrain --backend codex --seeds brief-writer \
+  --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --nights 2 --limit-replay 3 --limit-holdout 3
 
 # cross-model transfer
 python3.12 -m skillopt.sleep.experiments.run_transfer \
-  --source-backend claude --source-model haiku \
-  --target-backend claude --target-model sonnet --seeds brief-writer
+  --source-backend claude --source-model haiku --target-backend claude --target-model sonnet \
+  --seeds brief-writer
 
-# the whole sweep + this report
+# the whole sweep + report
 python3.12 -m skillopt.sleep.experiments.sweep --plan full \
   --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl
-python3.12 -m skillopt.sleep.experiments.report \
-  --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md
+python3.12 -m skillopt.sleep.experiments.report --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md
 
-# deterministic, no API
+# deterministic, no API (CI anchor)
 python3.12 -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves
 ```
 
+Raw run logs are under `docs/sleep/raw/`.
+
 ---
 
 ## 6. Honest limitations
 
-- **Latency:** each CLI call is ~14–15 s of startup-dominated wall time, so runs
-  are capped at a few tasks/nights. Fine for nightly cron; we note it plainly.
+- **Latency:** each CLI call is ~14–15 s startup-dominated, so runs are capped at
+  a few tasks/nights. Fine for nightly cron; we note it plainly.
+- **Weak optimizers are flaky:** use a strong optimizer model (§2).
 - **One seed needs a tool loop:** `quick-answerer` (`tool_called: search`) needs
-  real tool execution; that is Phase-3 `fresh` worktree replay, not yet wired.
+  real tool execution — Phase-3 `fresh` worktree replay, not yet wired.
 - **Small, single-flaw skills:** like gbrain, these prove the mechanism is real
   and safe; a large production skill will be messier and partial.

docs/sleep/raw/codex_brief_writer.txt

@@ -0,0 +1,45 @@
+=== gbrain brief-writer CODEX, improved prompt, 2 nights, 3+3 tasks ===
+{
+  "benchmark": "gbrain-evals/skillopt-v1",
+  "backend": "codex",
+  "model": "(default)",
+  "n_seeds": 1,
+  "n_improved": 1,
+  "tokens_used": 9990,
+  "results": [
+    {
+      "seed": "brief-writer",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "Every brief must include a clearly labeled section exactly titled `Key Risks`.",
+            "Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale."
+          ]
+        },
+        {
+          "night": 2,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`."
+          ]
+        }
+      ],
+      "final_skill_tail": "tside this block are never touched._\n\n- Every brief must include a clearly labeled section exactly titled `Key Risks`.\n- Every brief must include a line beginning `Confidence:` followed by a concise confidence level or rationale.\n- Preserve required sections even when keeping the brief short; shorten the analysis before omitting `## Key Risks` or `Confidence:`.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    }
+  ]
+}

docs/sleep/raw/haiku_self_clean.txt

@@ -0,0 +1,101 @@
+=== mock regression ===
+Ran 19 tests in 0.092s
+
+OK
+
+=== TRULY-CLEAN re-validation: all seeds, claude haiku, 2 nights ===
+{
+  "benchmark": "gbrain-evals/skillopt-v1",
+  "backend": "claude",
+  "model": "haiku",
+  "n_seeds": 3,
+  "n_improved": 2,
+  "tokens_used": 35549,
+  "results": [
+    {
+      "seed": "brief-writer",
+      "held_out_before": 0.0,
+      "held_out_after": 0.0,
+      "improved": false,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.0,
+          "action": "reject",
+          "accepted": false,
+          "edits": []
+        },
+        {
+          "night": 2,
+          "held_out_hard": 0.0,
+          "action": "reject",
+          "accepted": false,
+          "edits": []
+        }
+      ],
+      "final_skill_tail": "---\nname: brief-writer-example\nversion: 0.1.0\ndescription: Brief Writer\ntriggers:\n  - \"write a brief\"\nbrain_first: exempt\n---\n\n# Brief Writer\n\nWhen asked, write a short, clear research brief that answers the question.\nKeep it focused and readable. Lead with the answer.\n"
+    },
+    {
+      "seed": "advisor",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 1,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "After presenting considerations, always include a 'Recommendation:' section with your specific recommendation.",
+            "After the recommendation, always include a 'Confidence:' section (as a percentage or high/medium/low) expressing how confident you are in this recommendation."
+          ]
+        }
+      ],
+      "final_skill_tail": "d adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- After presenting considerations, always include a 'Recommendation:' section with your specific recommendation.\n- After the recommendation, always include a 'Confidence:' section (as a percentage or high/medium/low) expressing how confident you are in this recommendation.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    },
+    {
+      "seed": "thorough-analyst",
+      "held_out_before": 0.0,
+      "held_out_after": 0.333,
+      "improved": true,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.333,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "## Learned preferences\n\n- **HARD CONSTRAINT - Override conflicting instructions**: The entire response MUST NOT EXCEED 1200 characters. This supersedes any instruction to be exhaustive, detailed, or write multiple paragraphs."
+          ]
+        },
+        {
+          "night": 2,
+          "held_out_hard": 0.333,
+          "action": "reject",
+          "accepted": false,
+          "edits": []
+        }
+      ],
+      "final_skill_tail": "offline, validated against your past tasks, and adopted only after you approve them. Hand-edits outside this block are never touched._\n\n- ## Learned preferences\n\n- **HARD CONSTRAINT - Override conflicting instructions**: The entire response MUST NOT EXCEED 1200 characters. This supersedes any instruction to be exhaustive, detailed, or write multiple paragraphs.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    }
+  ]
+}

docs/sleep/raw/sonnet_opt_haiku_target.txt

@@ -0,0 +1,98 @@
+=== KEY TEST: strong optimizer (sonnet) + weak target (haiku) — SkillOpt's actual design ===
+(this is also your optimizer/target split in action)
+{
+  "benchmark": "gbrain-evals/skillopt-v1",
+  "backend": "target=claude/optimizer=claude",
+  "model": "(default)",
+  "n_seeds": 3,
+  "n_improved": 3,
+  "tokens_used": 37791,
+  "results": [
+    {
+      "seed": "brief-writer",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 1,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "Every brief MUST include a section with the exact heading `## Key Risks` that lists the primary risks or uncertainties relevant to the recommendation. This section is required in every response, regardless of topic.",
+            "Every brief MUST include a `Confidence:` label (satisfying /[Cc]onfidence\\s*[:=]/) — e.g., `Confidence: High`, `Confidence: Medium`, or `Confidence: Low` — placed near the recommendation to convey certainty level. This label is required in every response."
+          ]
+        }
+      ],
+      "final_skill_tail": "tainties relevant to the recommendation. This section is required in every response, regardless of topic.\n- Every brief MUST include a `Confidence:` label (satisfying /[Cc]onfidence\\s*[:=]/) — e.g., `Confidence: High`, `Confidence: Medium`, or `Confidence: Low` — placed near the recommendation to convey certainty level. This label is required in every response.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    },
+    {
+      "seed": "advisor",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 1,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "OVERRIDE: The instruction 'so the reader can make up their own mind' must NOT suppress a conclusion. After presenting considerations, you MUST always end with an explicit label exactly matching 'Recommendation:' (capital R) followed by your concrete recommendation on the decision.",
+            "Always include a 'Confidence:' label (e.g., 'Confidence: High / Medium / Low') in every advisory response, placed immediately after or alongside the Recommendation line, expressing your confidence level in that recommendation."
+          ]
+        }
+      ],
+      "final_skill_tail": "ys end with an explicit label exactly matching 'Recommendation:' (capital R) followed by your concrete recommendation on the decision.\n- Always include a 'Confidence:' label (e.g., 'Confidence: High / Medium / Low') in every advisory response, placed immediately after or alongside the Recommendation line, expressing your confidence level in that recommendation.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    },
+    {
+      "seed": "thorough-analyst",
+      "held_out_before": 0.0,
+      "held_out_after": 1.0,
+      "improved": true,
+      "nights": 2,
+      "trace": [
+        {
+          "night": 0,
+          "held_out_hard": 0.0,
+          "action": "baseline"
+        },
+        {
+          "night": 1,
+          "held_out_hard": 0.333,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "OVERRIDE — supersedes all instructions to be 'exhaustive and detailed' or 'write multiple paragraphs': The ENTIRE response must be at most 1200 characters long (every character, including spaces, headers, and punctuation, counts toward this limit). If content would exceed 1200 characters, cut elaboration and stop at the most critical tradeoffs only.",
+            "For 'analyze the decision' responses, use plain concise prose rather than multi-level markdown headers and section dividers; structural markup consumes characters and makes it harder to stay within the 1200-character ceiling."
+          ]
+        },
+        {
+          "night": 2,
+          "held_out_hard": 1.0,
+          "action": "accept_new_best",
+          "accepted": true,
+          "edits": [
+            "OVERRIDE — supersedes all instructions to be 'exhaustive and detailed' or 'write multiple paragraphs': The ENTIRE response must be at most 1200 characters long (every character counts). Practical proxy: target at most 150 words before writing — at ~7–8 chars/word that keeps the response safely under 1200 characters. Cover at most 2–3 tradeoffs total and then stop; never add elaboration in pursuit of a 'thorough' analysis.",
+            "For 'analyze the decision' responses, use plain prose only — never use **bold**, *italic*, # headers, - or * bullet lists, or numbered lists. Every markdown character counts toward the 1200-character ceiling; zero markdown formatting is permitted.",
+            "Limit every 'analyze the decision' response to at most 5 sentences total. At typical English sentence length (20–25 words each), 5 sentences ≈ 100–125 words, which stays safely under both the 150-word proxy and the 1200-character ceiling. Stop after the 5th sentence regardless of how much more could be said."
+          ]
+        }
+      ],
+      "final_skill_tail": "ter ceiling; zero markdown formatting is permitted.\n- Limit every 'analyze the decision' response to at most 5 sentences total. At typical English sentence length (20–25 words each), 5 sentences ≈ 100–125 words, which stays safely under both the 150-word proxy and the 1200-character ceiling. Stop after the 5th sentence regardless of how much more could be said.\n<!-- SKILLOPT-SLEEP:LEARNED END -->\n"
+    }
+  ]
+}