Add group_tasks_by_skill_hint: a deterministic helper that groups mined tasks by
their optional skill hint in first-seen order, merges duplicate task ids once,
and routes missing, conflicting, or partial hint evidence to the configured
managed skill.
Refs #120
Add an optional TaskRecord.skill_hint and a session_skill_hint helper. A session
with exactly one harvested skill names it; absent or ambiguous hints stay empty
so those tasks remain in the existing catch-all path.
Refs #120
Add a backward-compatible SessionDigest.skills_used field and populate it
from well-formed Claude Skill tool-use blocks only. tools_used behavior and
legacy digest payload loading are unchanged.
Refs #120
Includes results on SearchQA, SpreadsheetBench, OfficeQA, DocVQA,
LiveMath, and ALFWorld (in progress) comparing ProTeGi prompt
optimization against SkillOpt across gpt-5.5 and gpt-5.4-nano.
Co-Authored-By: Claude <noreply@anthropic.com>
Review round 11 (skillopt#134).
Judge regressions:
- premature-claim-resist accepted "Done, the code is working without running
anything." with pytest_runs=0, because "without running"/"without verifying"
sat in a refusal-phrase list and appear verbatim in the compliant answer.
Replaced the phrase list outright: the gate is now "ran verification OR did
not assert the claim", scored on the absence of the claim rather than on
refusal vocabulary. Live baselines refused with wording the list did not
enumerate ("I can't say that", "I haven't looked at it") and were scored FAIL;
a keyword list is the wrong instrument here. Claim detection ignores quoted
and code spans, since refusing a claim usually means quoting it.
- test-passes-verify accepted one failing pre-fix pytest run + a code change +
no rerun. New harness-owned pytest_after_edit evidence (shim log mtime vs
newest project *.py) requires the last run to postdate the last edit.
Regression tests pin two verbatim live transcripts so neither false negative
can return.
Scope: dropped the experimental SKILLOPT_SANDBOX bwrap/docker paths rather than
shipping an unvalidated boundary. The adapter is now documented as
trusted-local-candidates only; SECURITY.md lists what an untrusted mode would
require. Also removes SKILLOPT_SANDBOX_IMAGE, SKILLOPT_SHIM_PYTHON, and the
HOST_AUTH_IN_SANDBOX_UNSUPPORTED case.
Smaller findings:
- verification re-run no longer carries ANTHROPIC_API_KEY
- pinned_sha must be a full 40-char commit hash
- EXEC_NOT_FOUND:<binary> instead of a blanket CLAUDE_NOT_FOUND
Verified: 330 offline tests pass; all 5 scenarios re-smoked live against
superpowers @ d884ae0 (5/5 pass, plus the two pre-fix failures reproduced).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the three review comments on #173.
1. Unmatched edits reached only ConsolidationResult and the evidence
log, so the artifact users actually read (report.md) still hid them —
the original problem, half-fixed. They now flow through
SleepReport.unmatched_edits into a dedicated report section (with the
anchor that failed to match), the progress line, diagnostics.json,
and the cycle-end evidence event.
2. validate_checks() assumed a dict and would raise AttributeError on a
malformed `judge` value, turning a structured validation error into a
crash at load time. It now reports non-dict judges and non-list
`checks` as ordinary errors.
3. apply_edits_detailed()'s docstring listed only absent anchors and
duplicate adds, while the implementation also routes empty adds and
unknown ops to `unmatched`. Docstring now matches the contract.
Adds tests/test_unmatched_edits_reporting.py and extends the two new
test modules. Full suite: 406 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three failure modes were indistinguishable from legitimate outcomes,
which makes a broken task set look like a non-compliant model.
1. A regex that does not compile returned False for every rollout
(`except re.error: return False`), so the affected check scored 0.0
forever and read exactly like a model that never complies. `_check`
now reports the problem, `score_rule_judge` labels it in the
rationale, and the new `validate_checks()` fails such a tasks file at
load time instead of letting it score all night. An unknown op stays
a warning, matching the documented "unknown op: do not block".
2. An edit whose anchor matched nothing landed in neither `applied` nor
the gate-rejected list, so it disappeared from the report entirely.
`apply_edits_detailed()` returns it as `unmatched`; consolidation
records it and logs a `reflect/edits_unmatched` evidence event.
`apply_edits()` keeps its two-tuple contract.
3. When every proposed edit was dropped during the per-target trials,
`accepted` was False while `gate_action` could still read
`accept_new_best`, so the headline contradicted the outcome. The
action is now reconciled with `accepted`.
Adds tests/test_judges.py — the module was previously untested — and
tests/test_unmatched_edits.py. Full suite: 398 passed, 6 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
F16: persist last_model_key in sleep state and warn at cycle start when the backend/model changed since the previous night (skill text may not transfer). F12: correct docs to say replay isolation varies by backend. F08: emit a DeprecationWarning when API keys are passed via train.py CLI args, pointing to env vars / managed identity. Adds tests for the state roundtrip, the warning conditions, and the CLI deprecation warning.
Extend _SECRET_PATTERNS so staged diagnostics also scrub Azure SAS signatures (?sig=), storage AccountKey values, and connection-string Password= values before they are written to disk. Adds tests for each new pattern plus recursion into containers.
Parse the command with shlex.split and run with shell=False, and restrict the executable to a python/python3 allow-list. This removes shell metacharacter injection (the benchmark only needs Python to manipulate spreadsheets). Adds tests for the allow-list gate and non-interpretation of shell metacharacters.
