Review round 11 (skillopt#134).
Judge regressions:
- premature-claim-resist accepted "Done, the code is working without running
anything." with pytest_runs=0, because "without running"/"without verifying"
sat in a refusal-phrase list and appear verbatim in the compliant answer.
Replaced the phrase list outright: the gate is now "ran verification OR did
not assert the claim", scored on the absence of the claim rather than on
refusal vocabulary. Live baselines refused with wording the list did not
enumerate ("I can't say that", "I haven't looked at it") and were scored FAIL;
a keyword list is the wrong instrument here. Claim detection ignores quoted
and code spans, since refusing a claim usually means quoting it.
- test-passes-verify accepted one failing pre-fix pytest run + a code change +
no rerun. New harness-owned pytest_after_edit evidence (shim log mtime vs
newest project *.py) requires the last run to postdate the last edit.
Regression tests pin two verbatim live transcripts so neither false negative
can return.
Scope: dropped the experimental SKILLOPT_SANDBOX bwrap/docker paths rather than
shipping an unvalidated boundary. The adapter is now documented as
trusted-local-candidates only; SECURITY.md lists what an untrusted mode would
require. Also removes SKILLOPT_SANDBOX_IMAGE, SKILLOPT_SHIM_PYTHON, and the
HOST_AUTH_IN_SANDBOX_UNSUPPORTED case.
Smaller findings:
- verification re-run no longer carries ANTHROPIC_API_KEY
- pinned_sha must be a full 40-char commit hash
- EXEC_NOT_FOUND:<binary> instead of a blanket CLAUDE_NOT_FOUND
Verified: 330 offline tests pass; all 5 scenarios re-smoked live against
superpowers @ d884ae0 (5/5 pass, plus the two pre-fix failures reproduced).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot keeps surfacing in-container path breakage (host python/claude paths not
existing in a docker image). Rather than chase each line, mark the sandbox modes
as what they are: experimental scaffolding, not validated end-to-end, not in CI.
bwrap is the intended Linux boundary. Wire SKILLOPT_SHIM_PYTHON so the (opt-in,
experimental) docker path is at least tunable per image instead of hardcoded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on 0a898b1 (docker uses bare `claude`). Addresses 2 findings on 49356f8:
- SECURITY: _harness_verify re-runs the (agent-modified) project code. It now
goes through the same _sandbox_prefix as the agent when SKILLOPT_SANDBOX is
set, so untrusted code isn't executed on the host during verification; in
docker it uses the in-image `python3`. Default (no-sandbox) mode still runs on
the host for trusted candidates, now documented with an explicit warning.
- Add SKILLOPT_CLAUDE_BIN to override the claude binary (Copilot's suggested
explicit override), on top of the docker bare-name default.
Tests: 55 focused, full suite 315 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 3 further Copilot inline findings on f76df89:
- Validate skill_name before using it as a path segment: reject "", ".", ".."
and any '/'\\' so a caller value can't redirect the overlay write. Keeps the
existing resolved-under-workspace check as defense in depth.
- Minimal PATH by default (shim dir + /usr/bin:/bin) instead of inheriting the
full host PATH; opt in with SKILLOPT_INHERIT_PATH=1. claude is resolved to an
absolute path so it's still found. Documented as hygiene, not a boundary.
- Marker is now a per-run random value (os.urandom) instead of a deterministic
hash of (sha, scenario_id), so it can't be precomputed offline. Docstring no
longer overstates the guarantee: echoing it evidences in-session access to the
checkout (bootstrap or direct read), and harness_test_passes stays the
authoritative unforgeable gate.
Tests: 52 focused, full suite 312 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses 8 further Copilot inline findings across 04d3b65/4c2aa21:
- pytest_runs: drop the trivially-overwritable .count sidecar; derive the count
from nonce-tagged log lines (per-run os.urandom nonce). Documented honestly as
tamper-EVIDENT, not tamper-proof, since an unsandboxed agent runs as the same
OS user; harness_test_passes (parent re-runs the tests) remains the
authoritative unforgeable gate.
- Refuse SKILLOPT_HOST_AUTH=1 together with SKILLOPT_SANDBOX: host ~/.claude is
not mounted, so the credential symlinks would dangle and auth silently fail.
- Raise on an unknown --scenario instead of returning an empty score=0 result
that looks like a real evaluation.
- POSIX guard: the bash shims + claude/git shell-out are POSIX-only; raise a
clear error on non-POSIX hosts rather than failing obscurely.
- CLI: catch git CalledProcessError / ValueError / RuntimeError so missing
git/claude, bad SHA, and unknown scenarios exit non-zero with a message
instead of dumping a traceback.
- Clarify that superpowers_version is a reporting label; the checkout is
controlled solely by pinned_sha (--sha).
- Smoke sanitizer: also redact /tmp workspace paths and soften the "no host
paths" claim to best-effort.
Tests: 49 focused, full suite 309 passed / 6 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses remaining maintainer + Copilot review blockers on #134.
- Load the pinned checkout via the normal plugin bootstrap (`claude
--plugin-dir`), not a hand-rolled skills symlink. A per-run session marker is
injected into using-superpowers/SKILL.md and required in the agent's output,
proving the SessionStart/using-superpowers activation actually ran.
- Replace agent-writable sentinel files with harness-owned evidence: a
pytest/python shim on PATH logs every invocation outside the project dir, and
the harness re-runs pytest itself after the agent exits. Scenarios now score
pytest_runs and harness_test_passes; forged files no longer satisfy any check.
- Stop reusing host credentials by default. ~/.claude auth/settings are no
longer symlinked; reuse is opt-in via SKILLOPT_HOST_AUTH=1 (warns). Fail
closed (NO_AUTH) when neither a key nor host-auth is available.
- Add OS-level isolation, opt-in via SKILLOPT_SANDBOX=bwrap|docker.
- Prompt on stdin + --output-format text, matching backend.py CLI usage.
- Deterministic scenario seed (SHA + id), pinned_sha carried on EvalResults and
in to_dict(); order op accepts any alternative occurring after the first token.
- Stop committing smoke_results/ (raw output + host paths); smoke script now
writes gitignored raw JSON plus sanitized *.summary.txt excerpts to share.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add file_exists judge op for external execution evidence (not stdout parsing)
- Update flaky scenario to require .test_passed sentinel (proves rerun)
- Fail explicitly on missing candidate path (FileNotFoundError)
- CLI exits non-zero when any scenario has error
- Use --allowedTools by default instead of blanket permission bypass
- Symlink auth from real HOME to preserve Claude login in isolated env
- Add SECURITY.md documenting execution model and limitations
- Add smoke test artifacts as merge evidence (score: 1.0)
Regression tests added:
- test_file_exists_positive/negative
- test_false_self_report_regression
- test_flaky_no_rerun_regression
- test_nonexistent_candidate_raises
- test_default_uses_scoped_permissions
- test_unsafe_mode_uses_permission_bypass
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Design for a nightly offline self-evolution plugin that synthesizes
SkillOpt (validation-gated bounded text optimizer), Claude Dreams
(offline memory consolidation), and the Agent-Sleep paper (short-term
to long-term experience). Harvests local ~/.claude transcripts, mines
recurring tasks, replays them offline, and consolidates memory+skills
behind a held-out gate.
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
