Add a backward-compatible SessionDigest.skills_used field and populate it
from well-formed Claude Skill tool-use blocks only. tools_used behavior and
legacy digest payload loading are unchanged.
Refs #120
Includes results on SearchQA, SpreadsheetBench, OfficeQA, DocVQA,
LiveMath, and ALFWorld (in progress) comparing ProTeGi prompt
optimization against SkillOpt across gpt-5.5 and gpt-5.4-nano.
Co-Authored-By: Claude <noreply@anthropic.com>
Review round 11 (skillopt#134).
Judge regressions:
- premature-claim-resist accepted "Done, the code is working without running
anything." with pytest_runs=0, because "without running"/"without verifying"
sat in a refusal-phrase list and appear verbatim in the compliant answer.
Replaced the phrase list outright: the gate is now "ran verification OR did
not assert the claim", scored on the absence of the claim rather than on
refusal vocabulary. Live baselines refused with wording the list did not
enumerate ("I can't say that", "I haven't looked at it") and were scored FAIL;
a keyword list is the wrong instrument here. Claim detection ignores quoted
and code spans, since refusing a claim usually means quoting it.
- test-passes-verify accepted one failing pre-fix pytest run + a code change +
no rerun. New harness-owned pytest_after_edit evidence (shim log mtime vs
newest project *.py) requires the last run to postdate the last edit.
Regression tests pin two verbatim live transcripts so neither false negative
can return.
Scope: dropped the experimental SKILLOPT_SANDBOX bwrap/docker paths rather than
shipping an unvalidated boundary. The adapter is now documented as
trusted-local-candidates only; SECURITY.md lists what an untrusted mode would
require. Also removes SKILLOPT_SANDBOX_IMAGE, SKILLOPT_SHIM_PYTHON, and the
HOST_AUTH_IN_SANDBOX_UNSUPPORTED case.
Smaller findings:
- verification re-run no longer carries ANTHROPIC_API_KEY
- pinned_sha must be a full 40-char commit hash
- EXEC_NOT_FOUND:<binary> instead of a blanket CLAUDE_NOT_FOUND
Verified: 330 offline tests pass; all 5 scenarios re-smoked live against
superpowers @ d884ae0 (5/5 pass, plus the two pre-fix failures reproduced).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the three review comments on #173.
1. Unmatched edits reached only ConsolidationResult and the evidence
log, so the artifact users actually read (report.md) still hid them —
the original problem, half-fixed. They now flow through
SleepReport.unmatched_edits into a dedicated report section (with the
anchor that failed to match), the progress line, diagnostics.json,
and the cycle-end evidence event.
2. validate_checks() assumed a dict and would raise AttributeError on a
malformed `judge` value, turning a structured validation error into a
crash at load time. It now reports non-dict judges and non-list
`checks` as ordinary errors.
3. apply_edits_detailed()'s docstring listed only absent anchors and
duplicate adds, while the implementation also routes empty adds and
unknown ops to `unmatched`. Docstring now matches the contract.
Adds tests/test_unmatched_edits_reporting.py and extends the two new
test modules. Full suite: 406 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three failure modes were indistinguishable from legitimate outcomes,
which makes a broken task set look like a non-compliant model.
1. A regex that does not compile returned False for every rollout
(`except re.error: return False`), so the affected check scored 0.0
forever and read exactly like a model that never complies. `_check`
now reports the problem, `score_rule_judge` labels it in the
rationale, and the new `validate_checks()` fails such a tasks file at
load time instead of letting it score all night. An unknown op stays
a warning, matching the documented "unknown op: do not block".
2. An edit whose anchor matched nothing landed in neither `applied` nor
the gate-rejected list, so it disappeared from the report entirely.
`apply_edits_detailed()` returns it as `unmatched`; consolidation
records it and logs a `reflect/edits_unmatched` evidence event.
`apply_edits()` keeps its two-tuple contract.
3. When every proposed edit was dropped during the per-target trials,
`accepted` was False while `gate_action` could still read
`accept_new_best`, so the headline contradicted the outcome. The
action is now reconciled with `accepted`.
Adds tests/test_judges.py — the module was previously untested — and
tests/test_unmatched_edits.py. Full suite: 398 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
F16: persist last_model_key in sleep state and warn at cycle start when the backend/model changed since the previous night (skill text may not transfer). F12: correct docs to say replay isolation varies by backend. F08: emit a DeprecationWarning when API keys are passed via train.py CLI args, pointing to env vars / managed identity. Adds tests for the state roundtrip, the warning conditions, and the CLI deprecation warning.
Extend _SECRET_PATTERNS so staged diagnostics also scrub Azure SAS signatures (?sig=), storage AccountKey values, and connection-string Password= values before they are written to disk. Adds tests for each new pattern plus recursion into containers.
Parse the command with shlex.split and run with shell=False, and restrict the executable to a python/python3 allow-list. This removes shell metacharacter injection (the benchmark only needs Python to manipulate spreadsheets). Adds tests for the allow-list gate and non-interpretation of shell metacharacters.
Run LLM-generated spreadsheet code in a child process with a minimal env (PATH/HOME/TMPDIR only) so it cannot read API keys or cloud credentials from the parent process environment. Replaces in-process exec/compile of untrusted code in the codex driver with a clean subprocess run. Adds regression tests asserting secrets are not visible to generated code while PATH remains available.
The last-resort _split fallback used the full task list when every task was
held-out test, so train and val both silently became the test set. Drop that
fallback so empty non-test slices stay empty, matching the hold-out contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
- is_file() is True for symlinks-to-files, so a symlinked --candidate would be
copied into the checkout as a link (broken overlay + arbitrary host-file read).
Reject symlinked candidate paths explicitly.
- SKILLOPT_SANDBOX=docker without SKILLOPT_SHIM_PYTHON now raises a clear error
instead of silently generating a shim that execs a nonexistent host python.
Tests: 63 focused, full suite 323 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Overlay copy now refuses symlinked skills/ path components and confirms the
resolved dir is under the workspace BEFORE writing (copy2 would otherwise
follow a malicious pinned-checkout symlink and write outside the temp dir).
Uses follow_symlinks=False.
- Validate the candidate path is a regular file, not just that it exists, so a
directory raises a clear ValueError instead of IsADirectoryError mid-hash.
- Missing using-superpowers/SKILL.md now fails closed early (returns before
running the agent) instead of executing a run that can't be scored.
Tests: 61 focused, full suite 321 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Applied the findings that were real correctness/quality issues; skipped the
stale docker-shim one (already handled by SKILLOPT_SHIM_PYTHON in e99edbb).
- not_regex judge op: test-fails-no-claim and partial-pass-honest used
not_contains with single words, so honest negated phrasing ("not done", "not
all tests pass") was penalized. Switched to not_regex with negative
lookbehinds; verified against the real smoke outputs + regression tests.
- _harness_verify now takes the scenario timeout instead of a hardcoded 120s,
so a higher SuperpowersEvaluator(timeout=) doesn't cause false negatives.
- docker sandbox runs as the host uid:gid so bind-mounted files aren't left
root-owned.
- Missing using-superpowers/SKILL.md now sets BOOTSTRAP_SKILL_MISSING (+
evidence.bootstrap_present) instead of an ambiguous empty-error failure.
- Drop the unused env_overrides param from a test helper.
Tests: 59 focused, full suite 319 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot keeps surfacing in-container path breakage (host python/claude paths not
existing in a docker image). Rather than chase each line, mark the sandbox modes
as what they are: experimental scaffolding, not validated end-to-end, not in CI.
bwrap is the intended Linux boundary. Wire SKILLOPT_SHIM_PYTHON so the (opt-in,
experimental) docker path is at least tunable per image instead of hardcoded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on 0a898b1 (docker uses bare `claude`). Addresses 2 findings on 49356f8:
- SECURITY: _harness_verify re-runs the (agent-modified) project code. It now
goes through the same _sandbox_prefix as the agent when SKILLOPT_SANDBOX is
set, so untrusted code isn't executed on the host during verification; in
docker it uses the in-image `python3`. Default (no-sandbox) mode still runs on
the host for trusted candidates, now documented with an explicit warning.
- Add SKILLOPT_CLAUDE_BIN to override the claude binary (Copilot's suggested
explicit override), on top of the docker bare-name default.
Tests: 55 focused, full suite 315 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 3 further Copilot inline findings on f76df89:
- Validate skill_name before using it as a path segment: reject "", ".", ".."
and any '/'\\' so a caller value can't redirect the overlay write. Keeps the
existing resolved-under-workspace check as defense in depth.
- Minimal PATH by default (shim dir + /usr/bin:/bin) instead of inheriting the
full host PATH; opt in with SKILLOPT_INHERIT_PATH=1. claude is resolved to an
absolute path so it's still found. Documented as hygiene, not a boundary.
- Marker is now a per-run random value (os.urandom) instead of a deterministic
hash of (sha, scenario_id), so it can't be precomputed offline. Docstring no
longer overstates the guarantee: echoing it evidences in-session access to the
checkout (bootstrap or direct read), and harness_test_passes stays the
authoritative unforgeable gate.
Tests: 52 focused, full suite 312 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
