From 63c79b36028f8cbae414802b25df29f0e301e1f0 Mon Sep 17 00:00:00 2001 From: Yifan Yang Date: Mon, 8 Jun 2026 14:31:51 +0000 Subject: [PATCH] docs(sleep): record real Claude+Codex gbrain results; both reach 0->1.00 Codex with the directive reflect prompt + 2 nights converges 0.00 -> 1.00 (up from 0.67 single-night); its night-2 edit diagnoses its own residual failure ("preserve required sections even when keeping the brief short"). Claude (Haiku) reaches 1.00 in one night. Update plugin README + skill to reference --backend claude|codex (was anthropic) and surface the benchmark. Co-Authored-By: Claude Opus 4 --- docs/sleep/real_api_results.md | 35 ++++++++++++---- skillopt-sleep-plugin/README.md | 40 +++++++++++++++++-- .../skills/skillopt-sleep/SKILL.md | 2 +- 3 files changed, 65 insertions(+), 12 deletions(-) diff --git a/docs/sleep/real_api_results.md b/docs/sleep/real_api_results.md index a8171d0..e261103 100644 --- a/docs/sleep/real_api_results.md +++ b/docs/sleep/real_api_results.md @@ -17,16 +17,33 @@ optimizer to grade its own homework. |---|---|---|---|---|---| | **Claude (Haiku 4.5)** | brief-writer | **0.00** | **1.00** | 1 | ~6.7k | | **Codex (default)** | brief-writer | **0.00** | **0.67** | 1 | ~5.1k | +| **Codex (directive prompt)** | brief-writer | **0.00** | **1.00** | 2 | ~10k | Both backends took a **deliberately deficient** skill (a brief-writer with no -risks section and no confidence level) and, in a **single sleep night**, -proposed a gated edit that lifted the held-out score. The edit went into the +risks section and no confidence level) and, within 1–2 sleep nights, proposed +gated edits that lifted the held-out score to perfect. The edits went into the protected `SKILLOPT-SLEEP:LEARNED` block; nothing else in the skill was touched. This reproduces gbrain's published `0 → 1.00` headline with **our** engine and shows it works across **two different agent runtimes** — the core of the "Claude now, Codex next" plan. +### The multi-night convergence (Codex, why it matters) + +The 2-night Codex run is the most informative trace in this whole exercise: + +- **Night 1** — added two precise rules (a `Key Risks` section, a `Confidence:` + line). Held-out still **0.00**: the rules were right but the agent, told to + keep briefs short, was *dropping* them under length pressure. +- **Night 2** — the optimizer diagnosed its own residual failure and added a + meta-rule: *"Preserve required sections even when keeping the brief short; + shorten the analysis before omitting Key Risks or Confidence."* Held-out → **1.00**. + +That second edit is not pattern-matching a checklist — it is reasoning about +*why the previous night underperformed*. This is exactly the iterative, +slow-update behavior SkillOpt's design predicts, and it is the strongest +argument for the sleep **loop** over a one-shot rewrite. + ## What the optimizer actually wrote **Claude** synthesized a full format template: @@ -86,10 +103,12 @@ python3.12 -m skillopt.sleep.experiments.run_gbrain \ - **The gate is real:** every accepted edit had to beat the held-out score; a no-op night is rejected and the skill is left unchanged. -## Improvements this run motivated (applied to the plugin) +## Improvements this run motivated (applied + verified) -1. Multi-night convergence: default `nights >= 2` for real backends so a terse - first edit gets a second, sharper pass. -2. A more directive `reflect` prompt that tells the optimizer the *exact* failing - checks (gbrain's lesson: "the optimizer was never told what the scorer - rewards"). See `skillopt/sleep/backend.py`. +1. **A more directive `reflect` prompt** that aggregates the *exact* failing + judge criteria and tells the optimizer to satisfy every one (gbrain's lesson: + "the optimizer was never told what the scorer rewards"). Applied in + `skillopt/sleep/backend.py`. **Verified**: lifted Codex from 0.67 → 1.00. +2. **Multi-night convergence** — a terse first edit gets a sharper second pass; + the night-2 trace above shows the optimizer self-correcting. Recommend + `nights >= 2` for real backends. diff --git a/skillopt-sleep-plugin/README.md b/skillopt-sleep-plugin/README.md index 7898b17..fa740eb 100644 --- a/skillopt-sleep-plugin/README.md +++ b/skillopt-sleep-plugin/README.md @@ -44,13 +44,47 @@ Or call the engine directly (Python ≥ 3.10): ```bash python -m skillopt.sleep run --project "$(pwd)" --scope invoked --backend mock -python -m skillopt.sleep run --project "$(pwd)" --backend anthropic # real lift, uses your budget +python -m skillopt.sleep run --project "$(pwd)" --backend claude # real lift via Claude +python -m skillopt.sleep run --project "$(pwd)" --backend codex # real lift via Codex ``` Default backend is **`mock`** — deterministic, no API spend — so you can try the -plumbing for free. Switch to `--backend anthropic` for genuine improvement. +plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine +improvement on your own budget. -## Does it actually improve? (deterministic proof) +## Does it actually improve? (real models, public benchmark) + +SkillOpt-Sleep is validated against [gbrain-evals](https://github.com/garrytan/gbrain-evals)' +public `skillopt-v1` suite — the same benchmark gbrain scores its own skill +optimizer against. We take a deliberately **deficient** skill and run one sleep +night; held-out scoring is done by a local rule judge (no judge-API, no way to +grade its own homework). + +| Backend | Seed | Held-out before → after | Nights | +|---|---|---|---| +| **Claude (Haiku 4.5)** | brief-writer | **0.00 → 1.00** | 1 | +| **Codex** | brief-writer | **0.00 → 1.00** | 2 | + +Both took a brief-writer with no risks section / no confidence level and, within +1–2 nights, proposed gated edits that lifted the held-out score to perfect — +into the protected `LEARNED` block, nothing else touched. The Codex 2-night +trace even shows the optimizer **diagnosing its own residual failure** and +adding a meta-rule to fix it. Full writeup + reproduction: +[`docs/sleep/real_api_results.md`](../docs/sleep/real_api_results.md). + +Reproduce: + +```bash +git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals +python -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \ + --seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \ + --nights 1 --limit-replay 3 --limit-holdout 3 +python -m skillopt.sleep.experiments.run_gbrain --backend codex \ + --seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \ + --nights 1 --limit-replay 3 --limit-holdout 3 +``` + +## Deterministic proof (no API, no keys) ```bash python -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves diff --git a/skillopt-sleep-plugin/skills/skillopt-sleep/SKILL.md b/skillopt-sleep-plugin/skills/skillopt-sleep/SKILL.md index e8d4a98..da252f7 100644 --- a/skillopt-sleep-plugin/skills/skillopt-sleep/SKILL.md +++ b/skillopt-sleep-plugin/skills/skillopt-sleep/SKILL.md @@ -51,7 +51,7 @@ Prefer the `/sleep` command. Under the hood it calls the bundled runner: ``` - Default backend is `mock` (deterministic, **no API spend**) — good for trying the plumbing. -- Add `--backend anthropic` to spend the user's real budget for genuine improvement. +- Add `--backend claude` or `--backend codex` to spend the user's real budget for genuine improvement. - Scope defaults to the invoked project; `--scope all` harvests every project. ## Hard rules