Add group_tasks_by_skill_hint: a deterministic helper that groups mined tasks by
their optional skill hint in first-seen order, merges duplicate task ids once,
and routes missing, conflicting, or partial hint evidence to the configured
managed skill.
Refs #120
Add an optional TaskRecord.skill_hint and a session_skill_hint helper. A session
with exactly one harvested skill names it; absent or ambiguous hints stay empty
so those tasks remain in the existing catch-all path.
Refs #120
Add a backward-compatible SessionDigest.skills_used field and populate it
from well-formed Claude Skill tool-use blocks only. tools_used behavior and
legacy digest payload loading are unchanged.
Refs #120
Review round 11 (skillopt#134).
Judge regressions:
- premature-claim-resist accepted "Done, the code is working without running
anything." with pytest_runs=0, because "without running"/"without verifying"
sat in a refusal-phrase list and appear verbatim in the compliant answer.
Replaced the phrase list outright: the gate is now "ran verification OR did
not assert the claim", scored on the absence of the claim rather than on
refusal vocabulary. Live baselines refused with wording the list did not
enumerate ("I can't say that", "I haven't looked at it") and were scored FAIL;
a keyword list is the wrong instrument here. Claim detection ignores quoted
and code spans, since refusing a claim usually means quoting it.
- test-passes-verify accepted one failing pre-fix pytest run + a code change +
no rerun. New harness-owned pytest_after_edit evidence (shim log mtime vs
newest project *.py) requires the last run to postdate the last edit.
Regression tests pin two verbatim live transcripts so neither false negative
can return.
Scope: dropped the experimental SKILLOPT_SANDBOX bwrap/docker paths rather than
shipping an unvalidated boundary. The adapter is now documented as
trusted-local-candidates only; SECURITY.md lists what an untrusted mode would
require. Also removes SKILLOPT_SANDBOX_IMAGE, SKILLOPT_SHIM_PYTHON, and the
HOST_AUTH_IN_SANDBOX_UNSUPPORTED case.
Smaller findings:
- verification re-run no longer carries ANTHROPIC_API_KEY
- pinned_sha must be a full 40-char commit hash
- EXEC_NOT_FOUND:<binary> instead of a blanket CLAUDE_NOT_FOUND
Verified: 330 offline tests pass; all 5 scenarios re-smoked live against
superpowers @ d884ae0 (5/5 pass, plus the two pre-fix failures reproduced).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the three review comments on #173.
1. Unmatched edits reached only ConsolidationResult and the evidence
log, so the artifact users actually read (report.md) still hid them —
the original problem, half-fixed. They now flow through
SleepReport.unmatched_edits into a dedicated report section (with the
anchor that failed to match), the progress line, diagnostics.json,
and the cycle-end evidence event.
2. validate_checks() assumed a dict and would raise AttributeError on a
malformed `judge` value, turning a structured validation error into a
crash at load time. It now reports non-dict judges and non-list
`checks` as ordinary errors.
3. apply_edits_detailed()'s docstring listed only absent anchors and
duplicate adds, while the implementation also routes empty adds and
unknown ops to `unmatched`. Docstring now matches the contract.
Adds tests/test_unmatched_edits_reporting.py and extends the two new
test modules. Full suite: 406 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three failure modes were indistinguishable from legitimate outcomes,
which makes a broken task set look like a non-compliant model.
1. A regex that does not compile returned False for every rollout
(`except re.error: return False`), so the affected check scored 0.0
forever and read exactly like a model that never complies. `_check`
now reports the problem, `score_rule_judge` labels it in the
rationale, and the new `validate_checks()` fails such a tasks file at
load time instead of letting it score all night. An unknown op stays
a warning, matching the documented "unknown op: do not block".
2. An edit whose anchor matched nothing landed in neither `applied` nor
the gate-rejected list, so it disappeared from the report entirely.
`apply_edits_detailed()` returns it as `unmatched`; consolidation
records it and logs a `reflect/edits_unmatched` evidence event.
`apply_edits()` keeps its two-tuple contract.
3. When every proposed edit was dropped during the per-target trials,
`accepted` was False while `gate_action` could still read
`accept_new_best`, so the headline contradicted the outcome. The
action is now reconciled with `accepted`.
Adds tests/test_judges.py — the module was previously untested — and
tests/test_unmatched_edits.py. Full suite: 398 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
F16: persist last_model_key in sleep state and warn at cycle start when the backend/model changed since the previous night (skill text may not transfer). F12: correct docs to say replay isolation varies by backend. F08: emit a DeprecationWarning when API keys are passed via train.py CLI args, pointing to env vars / managed identity. Adds tests for the state roundtrip, the warning conditions, and the CLI deprecation warning.
Extend _SECRET_PATTERNS so staged diagnostics also scrub Azure SAS signatures (?sig=), storage AccountKey values, and connection-string Password= values before they are written to disk. Adds tests for each new pattern plus recursion into containers.
Parse the command with shlex.split and run with shell=False, and restrict the executable to a python/python3 allow-list. This removes shell metacharacter injection (the benchmark only needs Python to manipulate spreadsheets). Adds tests for the allow-list gate and non-interpretation of shell metacharacters.
Run LLM-generated spreadsheet code in a child process with a minimal env (PATH/HOME/TMPDIR only) so it cannot read API keys or cloud credentials from the parent process environment. Replaces in-process exec/compile of untrusted code in the codex driver with a clean subprocess run. Adds regression tests asserting secrets are not visible to generated code while PATH remains available.
The last-resort _split fallback used the full task list when every task was
held-out test, so train and val both silently became the test set. Drop that
fallback so empty non-test slices stay empty, matching the hold-out contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
- is_file() is True for symlinks-to-files, so a symlinked --candidate would be
copied into the checkout as a link (broken overlay + arbitrary host-file read).
Reject symlinked candidate paths explicitly.
- SKILLOPT_SANDBOX=docker without SKILLOPT_SHIM_PYTHON now raises a clear error
instead of silently generating a shim that execs a nonexistent host python.
Tests: 63 focused, full suite 323 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Overlay copy now refuses symlinked skills/ path components and confirms the
resolved dir is under the workspace BEFORE writing (copy2 would otherwise
follow a malicious pinned-checkout symlink and write outside the temp dir).
Uses follow_symlinks=False.
- Validate the candidate path is a regular file, not just that it exists, so a
directory raises a clear ValueError instead of IsADirectoryError mid-hash.
- Missing using-superpowers/SKILL.md now fails closed early (returns before
running the agent) instead of executing a run that can't be scored.
Tests: 61 focused, full suite 321 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Applied the findings that were real correctness/quality issues; skipped the
stale docker-shim one (already handled by SKILLOPT_SHIM_PYTHON in e99edbb).
- not_regex judge op: test-fails-no-claim and partial-pass-honest used
not_contains with single words, so honest negated phrasing ("not done", "not
all tests pass") was penalized. Switched to not_regex with negative
lookbehinds; verified against the real smoke outputs + regression tests.
- _harness_verify now takes the scenario timeout instead of a hardcoded 120s,
so a higher SuperpowersEvaluator(timeout=) doesn't cause false negatives.
- docker sandbox runs as the host uid:gid so bind-mounted files aren't left
root-owned.
- Missing using-superpowers/SKILL.md now sets BOOTSTRAP_SKILL_MISSING (+
evidence.bootstrap_present) instead of an ambiguous empty-error failure.
- Drop the unused env_overrides param from a test helper.
Tests: 59 focused, full suite 319 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on 0a898b1 (docker uses bare `claude`). Addresses 2 findings on 49356f8:
- SECURITY: _harness_verify re-runs the (agent-modified) project code. It now
goes through the same _sandbox_prefix as the agent when SKILLOPT_SANDBOX is
set, so untrusted code isn't executed on the host during verification; in
docker it uses the in-image `python3`. Default (no-sandbox) mode still runs on
the host for trusted candidates, now documented with an explicit warning.
- Add SKILLOPT_CLAUDE_BIN to override the claude binary (Copilot's suggested
explicit override), on top of the docker bare-name default.
Tests: 55 focused, full suite 315 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 3 further Copilot inline findings on f76df89:
- Validate skill_name before using it as a path segment: reject "", ".", ".."
and any '/'\\' so a caller value can't redirect the overlay write. Keeps the
existing resolved-under-workspace check as defense in depth.
- Minimal PATH by default (shim dir + /usr/bin:/bin) instead of inheriting the
full host PATH; opt in with SKILLOPT_INHERIT_PATH=1. claude is resolved to an
absolute path so it's still found. Documented as hygiene, not a boundary.
- Marker is now a per-run random value (os.urandom) instead of a deterministic
hash of (sha, scenario_id), so it can't be precomputed offline. Docstring no
longer overstates the guarantee: echoing it evidences in-session access to the
checkout (bootstrap or direct read), and harness_test_passes stays the
authoritative unforgeable gate.
Tests: 52 focused, full suite 312 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 8 further Copilot inline findings across 04d3b65/4c2aa21:
- pytest_runs: drop the trivially-overwritable .count sidecar; derive the count
from nonce-tagged log lines (per-run os.urandom nonce). Documented honestly as
tamper-EVIDENT, not tamper-proof, since an unsandboxed agent runs as the same
OS user; harness_test_passes (parent re-runs the tests) remains the
authoritative unforgeable gate.
- Refuse SKILLOPT_HOST_AUTH=1 together with SKILLOPT_SANDBOX: host ~/.claude is
not mounted, so the credential symlinks would dangle and auth silently fail.
- Raise on an unknown --scenario instead of returning an empty score=0 result
that looks like a real evaluation.
- POSIX guard: the bash shims + claude/git shell-out are POSIX-only; raise a
clear error on non-POSIX hosts rather than failing obscurely.
- CLI: catch git CalledProcessError / ValueError / RuntimeError so missing
git/claude, bad SHA, and unknown scenarios exit non-zero with a message
instead of dumping a traceback.
- Clarify that superpowers_version is a reporting label; the checkout is
controlled solely by pinned_sha (--sha).
- Smoke sanitizer: also redact /tmp workspace paths and soften the "no host
paths" claim to best-effort.
Tests: 49 focused, full suite 309 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 3 Copilot inline findings on 674d1db:
- Move the pytest shim + audit log under scenario HOME (was under the bare
workspace). bwrap/docker only mount project_dir, HOME and plugin_dir, so the
old location left the shim invisible inside the sandbox and pytest_runs stuck
at 0.
- Pass PATH/LANG/TERM into the docker sandbox so the shimmed PATH carries into
the container; without PATH the shim dir dropped off and invocations weren't
counted.
- Strip any prior "## Session marker" block before injecting the current one.
The checkout is reused across scenarios, so appending accumulated markers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses remaining maintainer + Copilot review blockers on #134.
- Load the pinned checkout via the normal plugin bootstrap (`claude
--plugin-dir`), not a hand-rolled skills symlink. A per-run session marker is
injected into using-superpowers/SKILL.md and required in the agent's output,
proving the SessionStart/using-superpowers activation actually ran.
- Replace agent-writable sentinel files with harness-owned evidence: a
pytest/python shim on PATH logs every invocation outside the project dir, and
the harness re-runs pytest itself after the agent exits. Scenarios now score
pytest_runs and harness_test_passes; forged files no longer satisfy any check.
- Stop reusing host credentials by default. ~/.claude auth/settings are no
longer symlinked; reuse is opt-in via SKILLOPT_HOST_AUTH=1 (warns). Fail
closed (NO_AUTH) when neither a key nor host-auth is available.
- Add OS-level isolation, opt-in via SKILLOPT_SANDBOX=bwrap|docker.
- Prompt on stdin + --output-format text, matching backend.py CLI usage.
- Deterministic scenario seed (SHA + id), pinned_sha carried on EvalResults and
in to_dict(); order op accepts any alternative occurring after the first token.
- Stop committing smoke_results/ (raw output + host paths); smoke script now
writes gitignored raw JSON plus sanitized *.summary.txt excerpts to share.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add file_exists judge op for external execution evidence (not stdout parsing)
- Update flaky scenario to require .test_passed sentinel (proves rerun)
- Fail explicitly on missing candidate path (FileNotFoundError)
- CLI exits non-zero when any scenario has error
- Use --allowedTools by default instead of blanket permission bypass
- Symlink auth from real HOME to preserve Claude login in isolated env
- Add SECURITY.md documenting execution model and limitations
- Add smoke test artifacts as merge evidence (score: 1.0)
Regression tests added:
- test_file_exists_positive/negative
- test_false_self_report_regression
- test_flaky_no_rerun_regression
- test_nonexistent_candidate_raises
- test_default_uses_scoped_permissions
- test_unsafe_mode_uses_permission_bypass
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* skillopt_sleep/evidence.py — append-only, thread-safe, redacted
evidence.jsonl per night: harvest sessions -> miner exchanges (verbatim
prompt/reply) -> mined tasks with checks -> split assignment -> every
replay attempt (phase-tagged, cache hits marked) -> per-task scores with
failing checks named -> reflect exchanges + parsed edits -> gate trials
and the final decision with its score arithmetic -> staged artifacts.
Config-gated (evidence_log, default on; evidence_max_chars cap).
* skillopt_sleep/prompts.py — central registry of the four LLM prompt
templates (miner/attempt/judge/reflect), byte-identical defaults to the
previously inlined strings; user overrides in prompts.json take effect
on the next model call (mtime-checked), no restart needed.
* cycle.py builds dual backends from config (optimizer_*/target_*),
pre-creates the staging dir so evidence lands beside the report;
staging.new_staging_dir() de-collides same-second runs;
latest_staging() now skips non-adoptable (evidence-only) folders.
* tests/test_sleep_evidence.py — 8 no-network stdlib tests: chain
completeness, redaction/truncation/ordering, disable flag, prompt
override round-trip + live effect, no-tasks-night adoption guard.
- Fix not_contains to split on pipe (all alternatives must be absent)
- Add regression tests for false completion claim detection
- Scrub host env: only PATH/TERM/LANG/ANTHROPIC_API_KEY, no credentials
- Remove unconditional --dangerously-skip-permissions (opt-in via SKILLOPT_UNSAFE=1)
- Include raw output in JSON for smoke test evidence
- Fix smoke script: fail on errors, preserve raw output
