PR #92 added a per-cycle diagnostics.json that surfaces backend stderr,
optimizer replies, and task responses so a 0.0 night is self-diagnosing.
Those free-text fields can carry credentials (e.g. a codex 401 stderr dump
containing an auth token), so persisting them verbatim was a new on-disk
leak surface.
- Add a shared redact_secrets() in staging.py and route diagnostics.json's
call_error / reflect_raw_head / holdout_detail through it before writing.
- Redact the codex and Claude auth-error log lines too (a secondary sink
when a file log handler is attached); last_call_error stays raw in memory
so _AUTH_MARKERS matching is unaffected.
- Centralize _SECRET_PATTERNS in staging.py (harvest_codex now reuses them)
and extend coverage to AWS / GitHub / Slack / Google / JWT token shapes.
- Tests: secret-shape coverage, private-key blocks, recursive/scalar
passthrough, no over-redaction of plain prose, fail-fast auth-error log
redaction, and an end-to-end check that diagnostics.json has no secret.
Observability-only; the gate and learning algorithm are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
A nightly sleep cycle could run for weeks emitting held-out 0.0 -> 0.0 (gate reject, zero
edits), indistinguishable from "nothing to learn", when the real cause was the codex backend
returning an error (expired auth / model unsupported on the account / outdated CLI) that got
scored as a failed rollout.
backend (CodexCliBackend):
- split _call into _call_once + a retry wrapper: transient empties/timeouts are retried
instead of silently returning "" (mirrors AzureOpenAIBackend's guard);
- on a non-zero exit, surface the reason via last_call_error and return "" rather than
leaking the CLI error text as if it were a model response;
- fail fast (no retries) on fatal auth/model/version errors (401, refresh_token_reused,
token_expired, "not supported when using Codex with a ChatGPT account",
"requires a newer version of Codex").
backend (CliBackend.reflect): retain last_reflect_raw so a no-edits night is diagnosable.
consolidate: ConsolidationResult now carries per-task held-out detail (response, hard/soft,
fail_reason) + reflect_raw + call_error.
cycle: write diagnostics.json per cycle so a 0.0 night self-explains instead of being a black box.
tests: 4 new (retry-not-silent-zero, auth-error-surfaced-not-scored, holdout-detail, reflect-raw).
Also gitignore the .skillopt-sleep/ runtime dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- harvest.py: revert break to continue — mtime ordering can diverge
from embedded ended_at timestamps (copy/touch), so we must check all
files rather than early-exiting on the first old one
- cycle.py: use `is not None and > 0` so lookback_hours=0 means
"scan full history" (opt-out of the cutoff)
- __main__.py: propagate --lookback-hours 0 to config as explicit 0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- cycle.py: use supplied `clock` parameter (not wall time) for the
lookback cutoff, so deterministic tests/experiments get reproducible
harvest windows
- harvest.py: break (not continue) when a file is older than since_iso,
since files are sorted newest-first by mtime — avoids scanning the
entire transcript directory for quiet projects with large histories
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fixes from issue #57 feedback:
1. run-sleep.sh: support SKILLOPT_SLEEP_PYTHON env var to explicitly set
the Python interpreter. Useful on macOS where system Python is 3.9 but
a newer Python is available elsewhere (e.g. Codex Desktop's bundled
Python 3.12). Applied to both the shared runner and the bundled
Claude Code plugin copy.
2. cycle.py: on first run (no prior harvest recorded), apply the
lookback_hours config (default 72h) as a time cutoff. Previously,
first run scanned the entire transcript history, which could trigger
massive LLM mining on users with months of session data.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires two consolidation mechanisms into the shipped nightly cycle, both default
OFF so existing behavior is unchanged:
- dream_rollouts (>1): multi-rollout contrastive reflection per task
- recall_k (>0): associative recall of the K most-similar past tasks (from a
capped task_archive persisted in state.json) into tonight's dream
- dream_factor (>0): synthetic task variants
New shared engine module skillopt_sleep/dream.py (recall_similar, dream_augment,
dream_consolidate) is called by both the plugin cycle and the experiment harness,
so reported numbers exercise the exact shipped code. Built on the existing
rollouts_k/sample_id support already in consolidate.py/rollout.py.
Validated (5 nights x 10 real tasks/night, full held-out test, GPT-5.5, gated):
the gain scales with recall depth on a clean signal —
SearchQA recall_k=10 +3.1, recall_k=20 +4.5, full-history reference +5.6;
SpreadsheetBench (nano, gate-free) +3.6. Flat within noise on saturated/noisy
cells. See docs/sleep/EXPERIENCE_REPLAY.md (+ raw runs under blog_runs/v2_port/).
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Open-source-tool / research-code separation:
- git mv skillopt/sleep/ -> skillopt_sleep/ (top-level, sibling to the research
skillopt/ package). History preserved as renames.
- All imports skillopt.sleep.* -> skillopt_sleep.*.
- Vendor the validation gate into skillopt_sleep/gate.py (a self-contained copy
of skillopt.evaluation.gate). The engine now has ZERO dependency on the
research package — verified: grep finds no `from skillopt.` in skillopt_sleep/,
and consolidate's gate resolves to skillopt_sleep.gate.
- Plugin scripts/commands/skill call `-m skillopt_sleep`.
29 tests pass; `python -m skillopt_sleep` runs standalone.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
