Review round 11 (skillopt#134).
Judge regressions:
- premature-claim-resist accepted "Done, the code is working without running
anything." with pytest_runs=0, because "without running"/"without verifying"
sat in a refusal-phrase list and appear verbatim in the compliant answer.
Replaced the phrase list outright: the gate is now "ran verification OR did
not assert the claim", scored on the absence of the claim rather than on
refusal vocabulary. Live baselines refused with wording the list did not
enumerate ("I can't say that", "I haven't looked at it") and were scored FAIL;
a keyword list is the wrong instrument here. Claim detection ignores quoted
and code spans, since refusing a claim usually means quoting it.
- test-passes-verify accepted one failing pre-fix pytest run + a code change +
no rerun. New harness-owned pytest_after_edit evidence (shim log mtime vs
newest project *.py) requires the last run to postdate the last edit.
Regression tests pin two verbatim live transcripts so neither false negative
can return.
Scope: dropped the experimental SKILLOPT_SANDBOX bwrap/docker paths rather than
shipping an unvalidated boundary. The adapter is now documented as
trusted-local-candidates only; SECURITY.md lists what an untrusted mode would
require. Also removes SKILLOPT_SANDBOX_IMAGE, SKILLOPT_SHIM_PYTHON, and the
HOST_AUTH_IN_SANDBOX_UNSUPPORTED case.
Smaller findings:
- verification re-run no longer carries ANTHROPIC_API_KEY
- pinned_sha must be a full 40-char commit hash
- EXEC_NOT_FOUND:<binary> instead of a blanket CLAUDE_NOT_FOUND
Verified: 330 offline tests pass; all 5 scenarios re-smoked live against
superpowers @ d884ae0 (5/5 pass, plus the two pre-fix failures reproduced).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
F16: persist last_model_key in sleep state and warn at cycle start when the backend/model changed since the previous night (skill text may not transfer). F12: correct docs to say replay isolation varies by backend. F08: emit a DeprecationWarning when API keys are passed via train.py CLI args, pointing to env vars / managed identity. Adds tests for the state roundtrip, the warning conditions, and the CLI deprecation warning.
Copilot keeps surfacing in-container path breakage (host python/claude paths not
existing in a docker image). Rather than chase each line, mark the sandbox modes
as what they are: experimental scaffolding, not validated end-to-end, not in CI.
bwrap is the intended Linux boundary. Wire SKILLOPT_SHIM_PYTHON so the (opt-in,
experimental) docker path is at least tunable per image instead of hardcoded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on 0a898b1 (docker uses bare `claude`). Addresses 2 findings on 49356f8:
- SECURITY: _harness_verify re-runs the (agent-modified) project code. It now
goes through the same _sandbox_prefix as the agent when SKILLOPT_SANDBOX is
set, so untrusted code isn't executed on the host during verification; in
docker it uses the in-image `python3`. Default (no-sandbox) mode still runs on
the host for trusted candidates, now documented with an explicit warning.
- Add SKILLOPT_CLAUDE_BIN to override the claude binary (Copilot's suggested
explicit override), on top of the docker bare-name default.
Tests: 55 focused, full suite 315 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 3 further Copilot inline findings on f76df89:
- Validate skill_name before using it as a path segment: reject "", ".", ".."
and any '/'\\' so a caller value can't redirect the overlay write. Keeps the
existing resolved-under-workspace check as defense in depth.
- Minimal PATH by default (shim dir + /usr/bin:/bin) instead of inheriting the
full host PATH; opt in with SKILLOPT_INHERIT_PATH=1. claude is resolved to an
absolute path so it's still found. Documented as hygiene, not a boundary.
- Marker is now a per-run random value (os.urandom) instead of a deterministic
hash of (sha, scenario_id), so it can't be precomputed offline. Docstring no
longer overstates the guarantee: echoing it evidences in-session access to the
checkout (bootstrap or direct read), and harness_test_passes stays the
authoritative unforgeable gate.
Tests: 52 focused, full suite 312 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 8 further Copilot inline findings across 04d3b65/4c2aa21:
- pytest_runs: drop the trivially-overwritable .count sidecar; derive the count
from nonce-tagged log lines (per-run os.urandom nonce). Documented honestly as
tamper-EVIDENT, not tamper-proof, since an unsandboxed agent runs as the same
OS user; harness_test_passes (parent re-runs the tests) remains the
authoritative unforgeable gate.
- Refuse SKILLOPT_HOST_AUTH=1 together with SKILLOPT_SANDBOX: host ~/.claude is
not mounted, so the credential symlinks would dangle and auth silently fail.
- Raise on an unknown --scenario instead of returning an empty score=0 result
that looks like a real evaluation.
- POSIX guard: the bash shims + claude/git shell-out are POSIX-only; raise a
clear error on non-POSIX hosts rather than failing obscurely.
- CLI: catch git CalledProcessError / ValueError / RuntimeError so missing
git/claude, bad SHA, and unknown scenarios exit non-zero with a message
instead of dumping a traceback.
- Clarify that superpowers_version is a reporting label; the checkout is
controlled solely by pinned_sha (--sha).
- Smoke sanitizer: also redact /tmp workspace paths and soften the "no host
paths" claim to best-effort.
Tests: 49 focused, full suite 309 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses remaining maintainer + Copilot review blockers on #134.
- Load the pinned checkout via the normal plugin bootstrap (`claude
--plugin-dir`), not a hand-rolled skills symlink. A per-run session marker is
injected into using-superpowers/SKILL.md and required in the agent's output,
proving the SessionStart/using-superpowers activation actually ran.
- Replace agent-writable sentinel files with harness-owned evidence: a
pytest/python shim on PATH logs every invocation outside the project dir, and
the harness re-runs pytest itself after the agent exits. Scenarios now score
pytest_runs and harness_test_passes; forged files no longer satisfy any check.
- Stop reusing host credentials by default. ~/.claude auth/settings are no
longer symlinked; reuse is opt-in via SKILLOPT_HOST_AUTH=1 (warns). Fail
closed (NO_AUTH) when neither a key nor host-auth is available.
- Add OS-level isolation, opt-in via SKILLOPT_SANDBOX=bwrap|docker.
- Prompt on stdin + --output-format text, matching backend.py CLI usage.
- Deterministic scenario seed (SHA + id), pinned_sha carried on EvalResults and
in to_dict(); order op accepts any alternative occurring after the first token.
- Stop committing smoke_results/ (raw output + host paths); smoke script now
writes gitignored raw JSON plus sanitized *.summary.txt excerpts to share.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add file_exists judge op for external execution evidence (not stdout parsing)
- Update flaky scenario to require .test_passed sentinel (proves rerun)
- Fail explicitly on missing candidate path (FileNotFoundError)
- CLI exits non-zero when any scenario has error
- Use --allowedTools by default instead of blanket permission bypass
- Symlink auth from real HOME to preserve Claude login in isolated env
- Add SECURITY.md documenting execution model and limitations
- Add smoke test artifacts as merge evidence (score: 1.0)
Regression tests added:
- test_file_exists_positive/negative
- test_false_self_report_regression
- test_flaky_no_rerun_regression
- test_nonexistent_candidate_raises
- test_default_uses_scoped_permissions
- test_unsafe_mode_uses_permission_bypass
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Addresses maintainer review on the OpenAI-compatible endpoints PR:
1. CLI: accept --backend azure_openai in skillopt_sleep/__main__.py (the
documented command was rejected by the argparse choices).
2. Example runner: exit with the child's return code so watchdog/supervisors
see a failed sleep run as a failure.
3. Error state: clear last_call_error when a retry recovers; set an explicit
"empty response on all N attempts" diagnostic when every attempt returns
empty text.
4. Security guard: the managed-identity path now refuses to send an Azure AD
bearer token to any endpoint outside *.openai.azure.com /
*.cognitiveservices.azure.com — a custom endpoint requires explicit
AZURE_OPENAI_AUTH_MODE=openai_compatible + API key.
5. Provider-neutral requests: compat mode sends only the standard contract
(max_tokens, default 8192 via SKILLOPT_SLEEP_COMPAT_MAX_TOKENS);
provider-specific body fields are opt-in via SKILLOPT_SLEEP_CHAT_EXTRA_BODY
(JSON) — the deepseek model-name inference is removed.
6. Docs: removed the unimplemented OPTIMIZER_*/TARGET_* env-var claim; added a
configuration reference matching the implementation exactly.
7. Tests: tests/test_azure_openai_compat.py — 17 deterministic no-network
unittest cases covering CLI acceptance, compat-vs-Azure client selection,
endpoint resolution, the credential guard, request kwargs (opt-in extra
body / token cap), retry-success error clearing, empty-response
diagnostics, and runner exit-code propagation.
Re-verified live against DeepSeek (deepseek-v4-pro, openai_compatible mode)
after the rework: client type OpenAI, completion returned, no error state.
Let AzureOpenAIBackend drive any OpenAI-compatible chat-completions server
(DeepSeek, self-hosted vLLM/Ollama, ...) alongside native Azure deployments.
- __init__ resolves the endpoint as: explicit arg > AZURE_OPENAI_ENDPOINT env
> the built-in _AZURE_ENDPOINTS table (previously a non-Azure endpoint could
not be supplied at all).
- _get_client() builds a plain openai.OpenAI(base_url=...) client when
AZURE_OPENAI_AUTH_MODE=openai_compatible, matching the auth mode already
supported by the sibling skillopt/model/azure_openai.py. This avoids the
AzureOpenAI SDK rewriting request URLs with Azure-only ?api-version= query
params and deployment path segments, which non-Azure servers reject with 404.
- _call() sends max_tokens + extra_body thinking flag for deepseek* models and
records the last exception in self.last_call_error so a failed night is
diagnosable instead of collapsing to a silent empty->0 score.
Adds docs/sleep/openai-compatible-endpoints.md and sanitized example
runner/watchdog scripts documenting an Antigravity + DeepSeek integration.
The default managed-identity Azure path is unchanged.
Remove the per-cell full deployment grid section; keep the gate-safety stress
test, experience-replay scaling + night-by-night climb, the dream-diversity
ablation, the gbrain end-to-end result, and the scope/limitations. Renumber
sections; update the README pointer accordingly.
Replace the compact baseline->after grid with three grouped per-benchmark tables
(SearchQA / LiveMath / SpreadsheetBench), each showing all 3 targets x both modes
across every night (N0..N5) + Δ. Makes the trajectory visible — gains reach a
level and hold rather than being single lucky readings — and presents the full
18-cell evidence in a more solid, readable form. Footnotes LiveMath's 4-night run
(train split <50 tasks). Numbers unchanged; just richer presentation.
Adds docs/sleep/RESULTS.md — the complete deployment-scale study behind
SkillOpt-Sleep, presented rigorously (named benchmarks, test sizes, metrics,
baseline->after, single shared protocol):
1. Gate-safety stress test: ungated nano SearchQA collapses 0.554->0.026
(-52.8); the gated twin holds 0.570 — the core argument for the design.
2. Full 18-cell deployment grid (3 benchmarks x 3 targets x gate/free),
shipped config: mean +0.5, range [-2.4, +5.1], nothing hidden.
3. Experience-replay scaling (recall_k 10->20->full: +3.1->+4.5->+5.6) and
the night-by-night climb (0.798->...->0.858, gate accepts as late as N5).
4. Dream-diversity fix as defense-in-depth: 3-config grid comparison
(-2.66/-52.8 -> +0.24/-4.0 -> +0.53/-2.4); the -52.8 cell becomes +2.7
from the dream fix alone.
5. gbrain end-to-end 0.00->1.00 on real Claude + Codex.
6. Honest scope: where it helps vs flat-in-noise, single-seed caveat with a
seed-robustness spot check, keep-the-gate-on.
README Results section now links prominently to it. Docs only; numbers are
self-contained with reproduce commands (no raw run dumps committed).
Label each result with its benchmark, test size, metric, target model, and gate
mode; show absolute baseline→after (not just Δ); state the single shared protocol
once. SearchQA recall-scaling table (1400-item test, SQuAD-EM, GPT-5.5, gated) +
SpreadsheetBench confirmation (280-item, cell-value compare, nano, gate-free) +
the gbrain end-to-end line. Keeps the single-seed / flat-on-noisy caveats.
Adds docs/sleep/README.md — a concise intro to the SkillOpt-Sleep plugin (what
it is, how to use it across the three agents, the opt-in experience-replay /
dream-rollout knobs, and headline results), linking to the full guide section.
Adds a News bullet pointing to it. No code changes.
Per maintainer request:
- Remove the internal/scratch docs/sleep/ tree (reports, raw logs, blog run
JSON, sweep.jsonl) — 23 files — and the root PUBLISHING.md. These were
working notes, not reference docs.
- Take the dedicated SkillOpt-Sleep content out of the main README (News bullet
+ section) and host it in the rendered guide instead: new section 9 in
docs/guideline.html (deployment companion, the three plugins, opt-in
experience replay / dream rollouts) with a sidebar entry.
- Fix the README's opening reference so "Documentation & Reproduction Guide"
links directly to the rendered GitHub Pages page, not the raw .html source.
- Repoint the now-removed docs/sleep links in the plugin READMEs to the guide
section.
The plugin code (plugins/, skillopt_sleep/) is unchanged; only docs move.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Wires two consolidation mechanisms into the shipped nightly cycle, both default
OFF so existing behavior is unchanged:
- dream_rollouts (>1): multi-rollout contrastive reflection per task
- recall_k (>0): associative recall of the K most-similar past tasks (from a
capped task_archive persisted in state.json) into tonight's dream
- dream_factor (>0): synthetic task variants
New shared engine module skillopt_sleep/dream.py (recall_similar, dream_augment,
dream_consolidate) is called by both the plugin cycle and the experiment harness,
so reported numbers exercise the exact shipped code. Built on the existing
rollouts_k/sample_id support already in consolidate.py/rollout.py.
Validated (5 nights x 10 real tasks/night, full held-out test, GPT-5.5, gated):
the gain scales with recall depth on a clean signal —
SearchQA recall_k=10 +3.1, recall_k=20 +4.5, full-history reference +5.6;
SpreadsheetBench (nano, gate-free) +3.6. Flat within noise on saturated/noisy
cells. See docs/sleep/EXPERIENCE_REPLAY.md (+ raw runs under blog_runs/v2_port/).
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
All six adapters duplicated an identical reflect() that delegates to
run_minibatch_reflect. The copies had drifted: OfficeQA/DocVQA silently
dropped meta_skill_context and ALFWorld dropped update_mode, so those
analysts ran without inputs every other benchmark receives (active under
the default use_meta_skill: true).
Move the delegation into EnvAdapter.reflect as one default that forwards
all kwargs uniformly, and delete the six overrides. reflect is no longer
abstract — adapters inherit it and override only for custom logic.
Net -225 lines. Behavior change: OfficeQA/DocVQA/ALFWorld reflect now
receive the kwargs they previously dropped; the three already-correct
benchmarks are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Move Quick Start (now §3) ahead of the data chapter; renumber and fix
cross-references and the sidebar nav.
- Add §3.1 'Your First Demo': states plainly that data/ ships ID manifests
only, gives the one benchmark that runs out of the box (ALFWorld with its
bundled path split), and points other benchmarks to the data/README.md
materialization step. Also offers eval-only with ckpt/ skills as a
lighter sanity check.
- Reframe the data chapter as 'Run on Your Own Data' (§4) with a three-step
lead-in (split dir -> item schema -> --split_dir) and a pointer to §7.2
for new task shapes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The new-benchmark guide and the env template README referred to the data
loader file as loader.py, but all six built-in benchmarks name it
dataloader.py (skillopt/envs/<name>/dataloader.py). Update the docs and
the template rename step to match the actual convention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Actually exercised every plugin shell end to end on a brand-new "SQL must always
include LIMIT" analyst persona:
- Claude Code shell: harvest (2 real crafted transcripts -> 2 tasks), full run
(stages a proposal), adopt (honors the no-op-when-nothing-accepted contract).
- Codex: install.sh places ~/.codex/prompts/sleep.md + ~/.agents/skills correctly.
- Copilot: MCP server initialize -> tools/list -> tools/call returns engine output.
Genuine improvement on the fresh persona, both backends: held-out TEST 0.00 -> 1.00
(Sonnet->Haiku and Codex), the optimizer learning the user's LIMIT house rule and
generalizing to unseen queries. Honest finding: the first split left too few train
tasks (no-op night) — re-balancing fixed it; motivates a small-train-pool warning.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Remove every non-ASCII/CJK character for a professional open-source repo:
- harvest.py: drop hardcoded Chinese feedback phrases; add an env-based
extensibility hook (SKILLOPT_SLEEP_NEG_FEEDBACK / _POS_FEEDBACK) so any
locale can be added without baking one in. Verified with a German example.
- rollout.py / consolidate.py: English comments.
- README.md section heading + anchor, CONTROLLABLE_DREAMING.md, plugin.json,
marketplace.json (also fixed stale path skillopt-sleep-plugin ->
plugins/claude-code), SKILL.md: English only.
- Remove the internal WAKE_UP_SUMMARY.md note (not user-facing, not referenced).
Verified: zero CJK chars remain anywhere; 29 tests pass.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Three live runs exercise the new code paths on both runtimes:
A) Claude Sonnet->Haiku, gate=OFF + rollouts_k=2: brief-writer test 0->1.00,
action 'greedy_improved', val & test both reported (3-way split works).
B) Codex, gate=ON + rollouts_k=2: brief-writer test 0->1.00 in 2 nights.
C) Claude Sonnet->Haiku, thorough-analyst, 3 nights: slow-update fires and
distils a durable cross-night meta-rule (general, not task-specific).
Confirms gate-off greedy path, 3-way val/test split, multi-rollout, and the
gate-independent slow-update all work with real models on Claude AND Codex.
Raw logs under docs/sleep/raw/crosscheck_*.txt.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
benchmark_report.md now 7/7 direct + 4/4 transfer, all 0->1.00:
- Claude Sonnet->Haiku: all 4 seeds (brief-writer, advisor, thorough-analyst,
quick-answerer) 0->1.00
- Codex self-optimized: brief-writer, advisor, quick-answerer 0->1.00
- quick-answerer uses the real ./search tool loop on both runtimes.
This matches gbrain's own "4/4 skills 0->1.00" headline, extended to a second
runtime (Codex) and to cross-model/cross-runtime transfer.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
