github/microsoft-SkillOpt
1
0
Fork 0
mirror of https://github.com/microsoft/SkillOpt.git synced 2026-07-08 08:59:22 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor(sleep): decouple engine to top-level skillopt_sleep/ (zero research dep)

Browse Source 
Open-source-tool / research-code separation:
  - git mv skillopt/sleep/ -> skillopt_sleep/ (top-level, sibling to the research
    skillopt/ package). History preserved as renames.
  - All imports skillopt.sleep.* -> skillopt_sleep.*.
  - Vendor the validation gate into skillopt_sleep/gate.py (a self-contained copy
    of skillopt.evaluation.gate). The engine now has ZERO dependency on the
    research package — verified: grep finds no `from skillopt.` in skillopt_sleep/,
    and consolidate's gate resolves to skillopt_sleep.gate.
  - Plugin scripts/commands/skill call `-m skillopt_sleep`.

29 tests pass; `python -m skillopt_sleep` runs standalone.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
This commit is contained in:
Yifan Yang
2026-06-08 14:31:52 +00:00
parent e2de84d36f
commit b02ffc2c99
32 changed files with 199 additions and 162 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
skillopt-sleep-plugin/README.md
View File

@@ -48,7 +48,7 @@ cd SkillOpt
```
The plugin's bundled runner (`scripts/sleep.sh`) auto-selects a Python ≥ 3.10
interpreter and calls the `skillopt.sleep` engine in the repo. No `pip install`
interpreter and calls the `skillopt_sleep` engine in the repo. No `pip install`
is required for the default `mock` backend or for `claude`/`codex` backends —
they shell out to the CLIs you already have.
@@ -65,9 +65,9 @@ they shell out to the CLIs you already have.
Or call the engine directly (Python ≥ 3.10):
```bash
python -m skillopt.sleep run --project "$(pwd)" --scope invoked --backend mock
python -m skillopt.sleep run --project "$(pwd)" --backend claude # real lift via Claude
python -m skillopt.sleep run --project "$(pwd)" --backend codex # real lift via Codex
python -m skillopt_sleep run --project "$(pwd)" --scope invoked --backend mock
python -m skillopt_sleep run --project "$(pwd)" --backend claude # real lift via Claude
python -m skillopt_sleep run --project "$(pwd)" --backend codex # real lift via Codex
```
Default backend is **`mock`** — deterministic, no API spend — so you can try the
@@ -98,10 +98,10 @@ Reproduce:
```bash
git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
python -m skillopt.sleep.experiments.run_gbrain --backend claude --model haiku \
python -m skillopt_sleep.experiments.run_gbrain --backend claude --model haiku \
--seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
--nights 1 --limit-replay 3 --limit-holdout 3
python -m skillopt.sleep.experiments.run_gbrain --backend codex \
python -m skillopt_sleep.experiments.run_gbrain --backend codex \
--seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
--nights 1 --limit-replay 3 --limit-holdout 3
```
@@ -109,8 +109,8 @@ python -m skillopt.sleep.experiments.run_gbrain --backend codex \
## Deterministic proof (no API, no keys)
```bash
python -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves
python -m skillopt.sleep.experiments.run_experiment --persona programmer --assert-improves
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves
```
Each prints the held-out score rising from baseline toward 1.0 as the gate

2
skillopt-sleep-plugin/commands/sleep.md
View File

@@ -18,7 +18,7 @@ held-out replay score, and nothing live is modified until the user adopts it.
## How to run it
The engine is the `skillopt.sleep` Python package in this repo. Use the
The engine is the `skillopt_sleep` Python package in this repo. Use the
**plugin's bundled runner** so the right interpreter and repo are on the path:
```bash

6
skillopt-sleep-plugin/scripts/sleep.sh
View File

@@ -1,12 +1,12 @@
#!/usr/bin/env bash
# SkillOpt-Sleep runner — invokes the skillopt.sleep engine with a suitable
# SkillOpt-Sleep runner — invokes the skillopt_sleep engine with a suitable
# Python interpreter, from the repo that contains this plugin.
#
# Usage: sleep.sh <run|dry-run|status|adopt|harvest> [extra args...]
set -euo pipefail
# Resolve the repo root: the plugin lives at <repo>/skillopt-sleep-plugin,
# so the engine package is at <repo>/skillopt/sleep. CLAUDE_PLUGIN_ROOT points
# so the engine package is at <repo>/skillopt_sleep. CLAUDE_PLUGIN_ROOT points
# at the plugin dir when run by Claude Code; fall back to this script's dir.
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
REPO_ROOT="$(cd "$PLUGIN_ROOT/.." && pwd)"
@@ -27,4 +27,4 @@ fi
if [ "$#" -eq 0 ]; then set -- status; fi
cd "$REPO_ROOT"
exec "$PY" -m skillopt.sleep "$@"
exec "$PY" -m skillopt_sleep "$@"

8
skillopt-sleep-plugin/skills/skillopt-sleep/SKILL.md
View File

@@ -1,6 +1,6 @@
---
name: skillopt-sleep
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like '让 agent 越用越好用', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule offline self-optimization. Drives the skillopt.sleep engine: harvest past sessions → mine recurring tasks → replay offline → consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like '让 agent 越用越好用', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions → mine recurring tasks → replay offline → consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
---
# SkillOpt-Sleep: offline self-evolution for a local Claude agent
@@ -62,7 +62,7 @@ Prefer the `/sleep` command. Under the hood it calls the bundled runner:
- Always show the user the **held-out baseline → candidate** score and the
exact proposed edits before suggesting adoption. Evidence before adoption.
- If asked whether it really helps, run
`python -m skillopt.sleep.experiments.run_experiment --persona researcher --json`
`python -m skillopt_sleep.experiments.run_experiment --persona researcher --json`
— a deterministic demo that proves held-out lift and that the gate blocks
harmful edits.
@@ -70,8 +70,8 @@ Prefer the `/sleep` command. Under the hood it calls the bundled runner:
```bash
# deterministic proof (no API): held-out score rises, gate blocks regressions
python -m skillopt.sleep.experiments.run_experiment --persona researcher --assert-improves
python -m skillopt.sleep.experiments.run_experiment --persona programmer --assert-improves
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves
```
See `docs/sleep/experiment_results.md` for recorded output and

4
skillopt/sleep/__init__.py → skillopt_sleep/__init__.py
View File

@@ -11,8 +11,8 @@ Synthesizes three ideas:
* Sleep — short-term experience -> long-term competence, offline
Public entry points:
* skillopt.sleep.cli — `python -m skillopt.sleep ...`
* skillopt.sleep.cycle.run_sleep_cycle(...)
* skillopt_sleep.cli — `python -m skillopt_sleep ...`
* skillopt_sleep.cycle.run_sleep_cycle(...)
"""
from __future__ import annotations

26
skillopt/sleep/__main__.py → skillopt_sleep/__main__.py
View File

@@ -1,10 +1,10 @@
"""SkillOpt-Sleep — command-line interface.
python -m skillopt.sleep run # full cycle: harvest->mine->replay->gate->stage
python -m skillopt.sleep dry-run # same but report only, no staging/adopt
python -m skillopt.sleep status # show state + latest staged proposal
python -m skillopt.sleep adopt # apply the latest staged proposal (with backup)
python -m skillopt.sleep harvest # just print what would be mined (debug)
python -m skillopt_sleep run # full cycle: harvest->mine->replay->gate->stage
python -m skillopt_sleep dry-run # same but report only, no staging/adopt
python -m skillopt_sleep status # show state + latest staged proposal
python -m skillopt_sleep adopt # apply the latest staged proposal (with backup)
python -m skillopt_sleep harvest # just print what would be mined (debug)
Common flags:
--project PATH project to evolve (default: cwd)
@@ -23,12 +23,12 @@ import os
import sys
from typing import Any, Dict
from skillopt.sleep.config import load_config
from skillopt.sleep.cycle import run_sleep_cycle
from skillopt.sleep.harvest import harvest
from skillopt.sleep.mine import mine
from skillopt.sleep.state import SleepState
from skillopt.sleep.staging import latest_staging, adopt as adopt_staging
from skillopt_sleep.config import load_config
from skillopt_sleep.cycle import run_sleep_cycle
from skillopt_sleep.harvest import harvest
from skillopt_sleep.mine import mine
from skillopt_sleep.state import SleepState
from skillopt_sleep.staging import latest_staging, adopt as adopt_staging
def _add_common(p: argparse.ArgumentParser) -> None:
@@ -90,7 +90,7 @@ def cmd_run(args, dry: bool = False) -> int:
if outcome.staging_dir:
print(f"[sleep] staged: {outcome.staging_dir}")
if not outcome.adopted:
print("[sleep] review it, then: python -m skillopt.sleep adopt")
print("[sleep] review it, then: python -m skillopt_sleep adopt")
if outcome.adopted:
print(f"[sleep] auto-adopted: {', '.join(outcome.adopted_paths)}")
return 0
@@ -164,7 +164,7 @@ def cmd_harvest(args) -> int:
def main(argv=None) -> int:
parser = argparse.ArgumentParser(prog="skillopt.sleep", description="SkillOpt-Sleep nightly self-evolution")
parser = argparse.ArgumentParser(prog="skillopt_sleep", description="SkillOpt-Sleep nightly self-evolution")
sub = parser.add_subparsers(dest="cmd", required=True)
p_run = sub.add_parser("run", help="run a full sleep cycle")

6
skillopt/sleep/backend.py → skillopt_sleep/backend.py
View File

@@ -26,7 +26,7 @@ import re
import subprocess
from typing import Any, Dict, List, Optional, Tuple
from skillopt.sleep.types import EditRecord, ReplayResult, TaskRecord
from skillopt_sleep.types import EditRecord, ReplayResult, TaskRecord
def skill_hash(content: str) -> str:
@@ -192,7 +192,7 @@ class MockBackend(Backend):
def judge(self, task: TaskRecord, response: str) -> Tuple[float, float, str]:
if task.reference_kind == "rule" and task.judge:
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
return score_rule_judge(task.judge, response)
if task.reference_kind == "exact" and task.reference:
hard = exact_score(task.reference, response)
@@ -303,7 +303,7 @@ class CliBackend(Backend):
def judge(self, task: TaskRecord, response: str) -> Tuple[float, float, str]:
# gbrain-style rule judge: scored locally, no API spend
if task.reference_kind == "rule" and task.judge:
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
return score_rule_judge(task.judge, response)
# exact references are scored locally — no API spend
if task.reference_kind == "exact" and task.reference:

0
skillopt/sleep/budget.py → skillopt_sleep/budget.py
View File

0
skillopt/sleep/config.py → skillopt_sleep/config.py
View File

35
skillopt/sleep/consolidate.py → skillopt_sleep/consolidate.py
View File

@@ -3,11 +3,8 @@
This is the core that makes nightly evolution *safe*: it proposes bounded
edits from replayed failures, applies them to a candidate skill/memory, then
**gates** the candidate on a held-out slice of the user's own tasks. Only a
candidate that strictly improves the held-out score is accepted — exactly the
SkillOpt validation gate, reused verbatim from ``skillopt.evaluation.gate``.
Reused from the main SkillOpt package (import-light, no `openai` needed):
* skillopt.evaluation.gate.evaluate_gate / select_gate_score
candidate that strictly improves the held-out score is accepted — the SkillOpt
validation gate, vendored self-contained in ``skillopt_sleep.gate``.
"""
from __future__ import annotations
@@ -15,26 +12,16 @@ import os
from dataclasses import dataclass
from typing import List, Optional, Tuple
from skillopt.sleep.backend import Backend
from skillopt.sleep.memory import apply_edits
from skillopt.sleep.replay import aggregate_scores, replay_batch
from skillopt.sleep.types import EditRecord, ReplayResult, TaskRecord
from skillopt_sleep.backend import Backend
from skillopt_sleep.memory import apply_edits
from skillopt_sleep.replay import aggregate_scores, replay_batch
from skillopt_sleep.types import EditRecord, ReplayResult, TaskRecord
# Reuse the real SkillOpt gate. This module imports cleanly without `openai`.
try:
from skillopt.evaluation.gate import evaluate_gate, select_gate_score
_HAVE_REPO_GATE = True
except Exception: # pragma: no cover - fallback keeps engine standalone
_HAVE_REPO_GATE = False
def select_gate_score(hard, soft, metric="hard", mixed_weight=0.5): # type: ignore
if metric == "hard":
return float(hard)
if metric == "soft":
return float(soft)
w = max(0.0, min(1.0, float(mixed_weight)))
return (1 - w) * float(hard) + w * float(soft)
# Self-contained validation gate (vendored from SkillOpt; zero dependency on the
# research package, so this open-source tool stays decoupled from the paper code).
from skillopt_sleep.gate import evaluate_gate, select_gate_score
_HAVE_REPO_GATE = True
@dataclass
@@ -140,7 +127,7 @@ def consolidate(
if rollouts_k > 1:
# multi-rollout contrastive reflection: run each train task K times
# and distill a rule from the good-vs-bad contrast (the "脑补" signal).
from skillopt.sleep.rollout import multi_rollout, contrastive_reflect
from skillopt_sleep.rollout import multi_rollout, contrastive_reflect
sets = [multi_rollout(backend, t, cand_skill, cand_memory, k=rollouts_k)
for t in train_tasks]
edits = contrastive_reflect(

20
skillopt/sleep/cycle.py → skillopt_sleep/cycle.py
View File

@@ -14,15 +14,15 @@ import time
from dataclasses import dataclass
from typing import Any, Dict, List, Optional
from skillopt.sleep.backend import get_backend
from skillopt.sleep.config import SleepConfig, load_config
from skillopt.sleep.consolidate import consolidate
from skillopt.sleep.harvest import harvest
from skillopt.sleep.memory import ensure_skill_scaffold
from skillopt.sleep.mine import mine
from skillopt.sleep.state import SleepState, _now_iso
from skillopt.sleep.staging import write_staging, adopt as adopt_staging
from skillopt.sleep.types import SessionDigest, SleepReport, TaskRecord
from skillopt_sleep.backend import get_backend
from skillopt_sleep.config import SleepConfig, load_config
from skillopt_sleep.consolidate import consolidate
from skillopt_sleep.harvest import harvest
from skillopt_sleep.memory import ensure_skill_scaffold
from skillopt_sleep.mine import mine
from skillopt_sleep.state import SleepState, _now_iso
from skillopt_sleep.staging import write_staging, adopt as adopt_staging
from skillopt_sleep.types import SessionDigest, SleepReport, TaskRecord
@dataclass
@@ -131,7 +131,7 @@ def run_sleep_cycle(
llm_miner = None
if cfg.get("backend", "mock") != "mock" and cfg.get("llm_mine", True):
try:
from skillopt.sleep.llm_miner import make_llm_miner
from skillopt_sleep.llm_miner import make_llm_miner
llm_miner = make_llm_miner(backend, max_tasks=cfg.get("max_tasks_per_night", 40))
except Exception:
llm_miner = None

0
skillopt/sleep/experiments/__init__.py → skillopt_sleep/experiments/__init__.py
View File

4
skillopt/sleep/experiments/gbrain_bench.py → skillopt_sleep/experiments/gbrain_bench.py
View File

@@ -17,7 +17,7 @@ We map:
judge -> TaskRecord.judge (+ reference_kind="rule")
This lets us reproduce gbrain's headline result with our engine and either the
claude or codex backend, scoring locally via skillopt.sleep.judges (no judge API).
claude or codex backend, scoring locally via skillopt_sleep.judges (no judge API).
"""
from __future__ import annotations
@@ -25,7 +25,7 @@ import json
import os
from typing import Dict, List, Optional, Tuple
from skillopt.sleep.types import TaskRecord
from skillopt_sleep.types import TaskRecord
SEED_DIRS = {

2
skillopt/sleep/experiments/personas.py → skillopt_sleep/experiments/personas.py
View File

@@ -12,7 +12,7 @@ from __future__ import annotations
from typing import List
from skillopt.sleep.types import TaskRecord
from skillopt_sleep.types import TaskRecord
def _t(i, intent, ref, rule, project="/personas/demo", outcome="fail") -> TaskRecord:

6
skillopt/sleep/experiments/report.py → skillopt_sleep/experiments/report.py
View File

@@ -1,7 +1,7 @@
"""SkillOpt-Sleep — turn a sweep JSONL into a presented Markdown scorecard.
Usage:
python -m skillopt.sleep.experiments.report --in docs/sleep/sweep.jsonl \
python -m skillopt_sleep.experiments.report --in docs/sleep/sweep.jsonl \
--out docs/sleep/benchmark_report.md
"""
from __future__ import annotations
@@ -101,9 +101,9 @@ def render(rows: List[Dict[str, Any]]) -> str:
out.append("")
out.append("```bash")
out.append("git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals")
out.append("python -m skillopt.sleep.experiments.sweep --plan full \\")
out.append("python -m skillopt_sleep.experiments.sweep --plan full \\")
out.append(" --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 --out docs/sleep/sweep.jsonl")
out.append("python -m skillopt.sleep.experiments.report \\")
out.append("python -m skillopt_sleep.experiments.report \\")
out.append(" --in docs/sleep/sweep.jsonl --out docs/sleep/benchmark_report.md")
out.append("```")
out.append("")

22
skillopt/sleep/experiments/run_experiment.py → skillopt_sleep/experiments/run_experiment.py
View File

@@ -14,9 +14,9 @@ What it proves:
the adopted artifact, re-scored, retains the lift.
Run:
python -m skillopt.sleep.experiments.run_experiment
python -m skillopt.sleep.experiments.run_experiment --persona programmer --nights 3
python -m skillopt.sleep.experiments.run_experiment --backend anthropic # real lift
python -m skillopt_sleep.experiments.run_experiment
python -m skillopt_sleep.experiments.run_experiment --persona programmer --nights 3
python -m skillopt_sleep.experiments.run_experiment --backend anthropic # real lift
"""
from __future__ import annotations
@@ -27,21 +27,21 @@ import sys
import tempfile
from typing import List
from skillopt.sleep.backend import get_backend
from skillopt.sleep.consolidate import consolidate
from skillopt.sleep.experiments.personas import (
from skillopt_sleep.backend import get_backend
from skillopt_sleep.consolidate import consolidate
from skillopt_sleep.experiments.personas import (
PERSONAS,
harmful_edit_task,
researcher_persona,
)
from skillopt.sleep.memory import ensure_skill_scaffold
from skillopt.sleep.replay import aggregate_scores, replay_batch
from skillopt.sleep.types import TaskRecord
from skillopt_sleep.memory import ensure_skill_scaffold
from skillopt_sleep.replay import aggregate_scores, replay_batch
from skillopt_sleep.types import TaskRecord
def _score_holdout(backend, tasks: List[TaskRecord], skill: str, memory: str,
metric: str = "mixed", w: float = 0.5) -> float:
from skillopt.sleep.consolidate import select_gate_score
from skillopt_sleep.consolidate import select_gate_score
# the persona experiment uses a 2-way split (train/val, no test); score on val
holdout = [t for t in tasks if t.split in ("val", "holdout")] or tasks
pairs = replay_batch(backend, holdout, skill, memory)
@@ -52,7 +52,7 @@ def _score_holdout(backend, tasks: List[TaskRecord], skill: str, memory: str,
def run(persona: str = "researcher", nights: int = 4, backend_name: str = "mock",
edit_budget: int = 4, seed: int = 42, model: str = "", codex_path: str = "",
limit_tasks: int = 0) -> dict:
from skillopt.sleep.mine import assign_splits
from skillopt_sleep.mine import assign_splits
make = PERSONAS.get(persona, researcher_persona)
items = make()

18
skillopt/sleep/experiments/run_gbrain.py → skillopt_sleep/experiments/run_gbrain.py
View File

@@ -13,9 +13,9 @@ Held-out scoring is done locally by the rule judge (no judge API). Only the
agent's `attempt` (and the optimizer's `reflect`) spend tokens.
Usage:
python -m skillopt.sleep.experiments.run_gbrain --backend mock
python -m skillopt.sleep.experiments.run_gbrain --backend claude --seeds brief-writer --nights 2
python -m skillopt.sleep.experiments.run_gbrain --backend codex --data-root /tmp/gbrain-evals/eval/data/skillopt-v1
python -m skillopt_sleep.experiments.run_gbrain --backend mock
python -m skillopt_sleep.experiments.run_gbrain --backend claude --seeds brief-writer --nights 2
python -m skillopt_sleep.experiments.run_gbrain --backend codex --data-root /tmp/gbrain-evals/eval/data/skillopt-v1
"""
from __future__ import annotations
@@ -24,14 +24,14 @@ import json
import sys
from typing import Dict, List, Optional
from skillopt.sleep.backend import build_backend, get_backend
from skillopt.sleep.consolidate import consolidate, select_gate_score
from skillopt.sleep.experiments.gbrain_bench import (
from skillopt_sleep.backend import build_backend, get_backend
from skillopt_sleep.consolidate import consolidate, select_gate_score
from skillopt_sleep.experiments.gbrain_bench import (
available_seeds,
find_data_root,
load_seed,
)
from skillopt.sleep.replay import aggregate_scores, replay_batch
from skillopt_sleep.replay import aggregate_scores, replay_batch
def _score(backend, tasks, skill, memory, split="test", metric="mixed", w=0.5):
@@ -95,7 +95,7 @@ def run_seed(backend, seed: str, skill: str, tasks: List, *,
slow_text = None
if nights >= 2 and slow_update:
try:
from skillopt.sleep.slow_update import run_slow_update, replace_slow_field
from skillopt_sleep.slow_update import run_slow_update, replace_slow_field
val_tasks = [t for t in tasks if t.split == "val"] or tasks
prev_pairs = replay_batch(backend, val_tasks, first_night_skill, memory)
curr_pairs = replay_batch(backend, val_tasks, cur, memory)
@@ -170,7 +170,7 @@ def main(argv=None) -> int:
# budget auto-planning: derive nights x rollouts_k from a token budget
nights, rollouts_k = args.nights, args.rollouts_k
if args.budget_tokens:
from skillopt.sleep.budget import Budget, plan_depth
from skillopt_sleep.budget import Budget, plan_depth
n_train = len([t for t in tasks if t.split == "train"]) or len(tasks)
nights, rollouts_k = plan_depth(
Budget(max_tokens=args.budget_tokens), n_tasks=n_train,

10
skillopt/sleep/experiments/run_transfer.py → skillopt_sleep/experiments/run_transfer.py
View File

@@ -16,7 +16,7 @@ Protocol, per gbrain seed:
Report baseline / direct / transferred, mirroring SkillOpt Table "transfer".
Usage:
python -m skillopt.sleep.experiments.run_transfer \
python -m skillopt_sleep.experiments.run_transfer \
--source-backend claude --source-model haiku \
--target-backend claude --target-model sonnet \
--seeds brief-writer --nights 2
@@ -28,12 +28,12 @@ import json
import sys
from typing import List, Optional
from skillopt.sleep.backend import get_backend
from skillopt.sleep.consolidate import consolidate, select_gate_score
from skillopt.sleep.experiments.gbrain_bench import (
from skillopt_sleep.backend import get_backend
from skillopt_sleep.consolidate import consolidate, select_gate_score
from skillopt_sleep.experiments.gbrain_bench import (
available_seeds, find_data_root, load_seed,
)
from skillopt.sleep.replay import aggregate_scores, replay_batch
from skillopt_sleep.replay import aggregate_scores, replay_batch
def _holdout_hard(backend, tasks, skill, memory="") -> float:

12
skillopt/sleep/experiments/sweep.py → skillopt_sleep/experiments/sweep.py
View File

@@ -8,8 +8,8 @@ survive) and resume (skip configs whose row already exists).
Then `report.py` turns the JSONL into a presented Markdown scorecard.
Usage:
python -m skillopt.sleep.experiments.sweep --plan quick --out docs/sleep/sweep.jsonl
python -m skillopt.sleep.experiments.sweep --plan full --out docs/sleep/sweep.jsonl
python -m skillopt_sleep.experiments.sweep --plan quick --out docs/sleep/sweep.jsonl
python -m skillopt_sleep.experiments.sweep --plan full --out docs/sleep/sweep.jsonl
"""
from __future__ import annotations
@@ -20,10 +20,10 @@ import sys
import time
from typing import Any, Dict, List
from skillopt.sleep.backend import build_backend, get_backend
from skillopt.sleep.experiments.gbrain_bench import find_data_root, load_seed
from skillopt.sleep.experiments.run_gbrain import run_seed as bench_seed
from skillopt.sleep.experiments.run_transfer import run_seed as transfer_seed
from skillopt_sleep.backend import build_backend, get_backend
from skillopt_sleep.experiments.gbrain_bench import find_data_root, load_seed
from skillopt_sleep.experiments.run_gbrain import run_seed as bench_seed
from skillopt_sleep.experiments.run_transfer import run_seed as transfer_seed
# Plans: lists of config dicts. Kept small per-run to bound cost/latency.

50
skillopt_sleep/gate.py Normal file
View File

@@ -0,0 +1,50 @@
"""SkillOpt-Sleep — vendored validation gate.
This is a self-contained copy of the SkillOpt validation gate so the sleep
engine has ZERO dependency on the research package (skillopt/*). The research
repo's ``skillopt.evaluation.gate`` is the reference implementation and the two
are kept behaviourally identical; vendoring keeps this open-source tool
decoupled from the paper's experiment code.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class GateResult:
action: str # "accept_new_best" | "accept" | "reject"
current_skill: str
current_score: float
best_skill: str
best_score: float
best_step: int
def select_gate_score(hard: float, soft: float, metric: str = "hard",
mixed_weight: float = 0.5) -> float:
"""Project (hard, soft) onto a single comparison metric."""
if metric == "hard":
return float(hard)
if metric == "soft":
return float(soft)
if metric == "mixed":
w = max(0.0, min(1.0, float(mixed_weight)))
return (1.0 - w) * float(hard) + w * float(soft)
raise ValueError(f"unknown gate metric {metric!r}; expected hard/soft/mixed")
def evaluate_gate(candidate_skill: str, cand_hard: float, current_skill: str,
current_score: float, best_skill: str, best_score: float,
best_step: int, global_step: int, *, cand_soft: float = 0.0,
metric: str = "hard", mixed_weight: float = 0.5) -> GateResult:
"""Pure gate decision: compare candidate score to current/best."""
cand_score = select_gate_score(cand_hard, cand_soft, metric, mixed_weight)
if cand_score > current_score:
if cand_score > best_score:
return GateResult("accept_new_best", candidate_skill, cand_score,
candidate_skill, cand_score, global_step)
return GateResult("accept", candidate_skill, cand_score,
best_skill, best_score, best_step)
return GateResult("reject", current_skill, current_score,
best_skill, best_score, best_step)

2
skillopt/sleep/harvest.py → skillopt_sleep/harvest.py
View File

@@ -19,7 +19,7 @@ import json
import os
from typing import Any, Dict, Iterable, List, Optional
from skillopt.sleep.types import SessionDigest
from skillopt_sleep.types import SessionDigest
# Heuristic phrases that signal the user (dis)approving of prior output.

0
skillopt/sleep/judges.py → skillopt_sleep/judges.py
View File

4
skillopt/sleep/llm_miner.py → skillopt_sleep/llm_miner.py
View File

@@ -22,8 +22,8 @@ import json
import re
from typing import Any, Callable, Dict, List
from skillopt.sleep.backend import Backend, _extract_json
from skillopt.sleep.types import SessionDigest, TaskRecord
from skillopt_sleep.backend import Backend, _extract_json
from skillopt_sleep.types import SessionDigest, TaskRecord
_MINER_PROMPT = """You are mining a user's past AI-assistant sessions to find RECURRING tasks

2
skillopt/sleep/memory.py → skillopt_sleep/memory.py
View File

@@ -10,7 +10,7 @@ from __future__ import annotations
import re
from typing import List, Tuple
from skillopt.sleep.types import EditRecord
from skillopt_sleep.types import EditRecord
LEARNED_START = "<!-- SKILLOPT-SLEEP:LEARNED START -->"

2
skillopt/sleep/mine.py → skillopt_sleep/mine.py
View File

@@ -18,7 +18,7 @@ import hashlib
import re
from typing import Any, Callable, List, Optional
from skillopt.sleep.types import SessionDigest, TaskRecord
from skillopt_sleep.types import SessionDigest, TaskRecord
def _tid(project: str, intent: str) -> str:

6
skillopt/sleep/replay.py → skillopt_sleep/replay.py
View File

@@ -11,8 +11,8 @@ from __future__ import annotations
from typing import List, Tuple
from skillopt.sleep.backend import Backend
from skillopt.sleep.types import ReplayResult, TaskRecord
from skillopt_sleep.backend import Backend
from skillopt_sleep.types import ReplayResult, TaskRecord
def _required_tools(task: TaskRecord) -> List[str]:
@@ -44,7 +44,7 @@ def replay_one(backend: Backend, task: TaskRecord, skill: str, memory: str) -> R
# rule judges may need the detected tool calls; score locally when possible
if task.reference_kind == "rule" and task.judge:
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
hard, soft, rationale = score_rule_judge(task.judge, response, tools_called)
else:
hard, soft, rationale = backend.judge(task, response)

6
skillopt/sleep/rollout.py → skillopt_sleep/rollout.py
View File

@@ -18,9 +18,9 @@ from __future__ import annotations
from dataclasses import dataclass, field
from typing import List, Optional, Tuple
from skillopt.sleep.backend import Backend, _extract_json
from skillopt.sleep.replay import replay_one
from skillopt.sleep.types import EditRecord, ReplayResult, TaskRecord
from skillopt_sleep.backend import Backend, _extract_json
from skillopt_sleep.replay import replay_one
from skillopt_sleep.types import EditRecord, ReplayResult, TaskRecord
@dataclass

4
skillopt/sleep/slow_update.py → skillopt_sleep/slow_update.py
View File

@@ -23,8 +23,8 @@ from __future__ import annotations
import re
from typing import List, Optional, Tuple
from skillopt.sleep.backend import Backend, _extract_json
from skillopt.sleep.types import ReplayResult, TaskRecord
from skillopt_sleep.backend import Backend, _extract_json
from skillopt_sleep.types import ReplayResult, TaskRecord
SLOW_UPDATE_START = "<!-- SLOW_UPDATE_START -->"

2
skillopt/sleep/staging.py → skillopt_sleep/staging.py
View File

@@ -13,7 +13,7 @@ import shutil
import time
from typing import List, Optional
from skillopt.sleep.types import SleepReport
from skillopt_sleep.types import SleepReport
def _ts_dir() -> str:

0
skillopt/sleep/state.py → skillopt_sleep/state.py
View File

2
skillopt/sleep/types.py → skillopt_sleep/types.py
View File

@@ -18,7 +18,7 @@ from typing import Any, Dict, List, Optional
class SessionDigest:
"""A normalized summary of one Claude Code session transcript.
Produced by :mod:`skillopt.sleep.harvest` from a ``<sessionId>.jsonl``
Produced by :mod:`skillopt_sleep.harvest` from a ``<sessionId>.jsonl``
transcript plus ``history.jsonl`` entries.
"""

84
tests/test_sleep_engine.py
View File

@@ -2,7 +2,7 @@
Pure-stdlib (unittest), deterministic, no API key, no third-party deps.
Run: python3.12 -m pytest tests/test_sleep_engine.py
or: python3.12 -m unittest skillopt.sleep ... (see bottom)
or: python3.12 -m unittest skillopt_sleep ... (see bottom)
"""
from __future__ import annotations
@@ -11,16 +11,16 @@ import os
import tempfile
import unittest
from skillopt.sleep.backend import MockBackend, exact_score, keyword_soft_score
from skillopt.sleep.config import load_config
from skillopt.sleep.consolidate import consolidate
from skillopt.sleep.cycle import run_sleep_cycle
from skillopt.sleep.experiments.personas import researcher_persona, programmer_persona
from skillopt.sleep.harvest import digest_transcript, _detect_feedback, _is_meta_prompt
from skillopt.sleep.memory import apply_edits, current_learned_lines, extract_learned, set_learned
from skillopt.sleep.mine import assign_splits, heuristic_mine, dedup_tasks
from skillopt.sleep.staging import adopt, latest_staging
from skillopt.sleep.types import EditRecord, SessionDigest, TaskRecord
from skillopt_sleep.backend import MockBackend, exact_score, keyword_soft_score
from skillopt_sleep.config import load_config
from skillopt_sleep.consolidate import consolidate
from skillopt_sleep.cycle import run_sleep_cycle
from skillopt_sleep.experiments.personas import researcher_persona, programmer_persona
from skillopt_sleep.harvest import digest_transcript, _detect_feedback, _is_meta_prompt
from skillopt_sleep.memory import apply_edits, current_learned_lines, extract_learned, set_learned
from skillopt_sleep.mine import assign_splits, heuristic_mine, dedup_tasks
from skillopt_sleep.staging import adopt, latest_staging
from skillopt_sleep.types import EditRecord, SessionDigest, TaskRecord
class TestScoring(unittest.TestCase):
@@ -115,7 +115,7 @@ class TestMine(unittest.TestCase):
def test_dream_never_in_val_or_test(self):
# the anti-overfitting guarantee: origin='dream' tasks only ever land in train
from skillopt.sleep.types import TaskRecord
from skillopt_sleep.types import TaskRecord
real = researcher_persona()
dream = [TaskRecord(id=f"d{i}", project="/p", intent=f"dream {i}",
origin="dream", derived_from="r0") for i in range(5)]
@@ -152,7 +152,7 @@ class TestConsolidateGate(unittest.TestCase):
class TestRuleJudge(unittest.TestCase):
def test_section_and_regex(self):
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
j = {"kind": "rule", "checks": [
{"op": "section_present", "arg": "Key Risks"},
{"op": "regex", "arg": r"[Cc]onfidence\s*[:=]"},
@@ -162,13 +162,13 @@ class TestRuleJudge(unittest.TestCase):
self.assertEqual(score_rule_judge(j, "just an answer")[0], 0.0)
def test_max_chars(self):
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
j = {"checks": [{"op": "max_chars", "arg": 50}]}
self.assertEqual(score_rule_judge(j, "x" * 10)[0], 1.0)
self.assertEqual(score_rule_judge(j, "x" * 100)[0], 0.0)
def test_partial_soft_score(self):
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
j = {"checks": [
{"op": "contains", "arg": "alpha"},
{"op": "contains", "arg": "beta"},
@@ -180,7 +180,7 @@ class TestRuleJudge(unittest.TestCase):
class TestGbrainLoader(unittest.TestCase):
def test_loads_when_present(self):
from skillopt.sleep.experiments.gbrain_bench import find_data_root, load_seed
from skillopt_sleep.experiments.gbrain_bench import find_data_root, load_seed
root = find_data_root()
if not root:
self.skipTest("gbrain-evals data not present")
@@ -191,7 +191,7 @@ class TestGbrainLoader(unittest.TestCase):
self.assertTrue(any(t.split == "val" for t in tasks))
self.assertTrue(all(t.reference_kind == "rule" for t in tasks))
# the deficient skill must FAIL its own held-out (test) checks (baseline 0)
from skillopt.sleep.judges import score_rule_judge
from skillopt_sleep.judges import score_rule_judge
ho = [t for t in tasks if t.split == "test"][0]
self.assertEqual(score_rule_judge(ho.judge, skill)[0], 0.0)
@@ -199,8 +199,8 @@ class TestGbrainLoader(unittest.TestCase):
class TestLlmMiner(unittest.TestCase):
def test_miner_emits_checkable_tasks(self):
# a stub backend whose _call returns canned miner JSON => deterministic
from skillopt.sleep.backend import Backend
from skillopt.sleep.llm_miner import make_llm_miner
from skillopt_sleep.backend import Backend
from skillopt_sleep.llm_miner import make_llm_miner
class StubBackend(Backend):
name = "stub"
@@ -219,8 +219,8 @@ class TestLlmMiner(unittest.TestCase):
self.assertEqual(tasks[0].judge["checks"][0]["op"], "section_present")
def test_miner_drops_uncheckable(self):
from skillopt.sleep.backend import Backend
from skillopt.sleep.llm_miner import make_llm_miner
from skillopt_sleep.backend import Backend
from skillopt_sleep.llm_miner import make_llm_miner
class EmptyBackend(Backend):
name = "stub"
@@ -234,8 +234,8 @@ class TestLlmMiner(unittest.TestCase):
class TestMultiObjectiveAndPrefs(unittest.TestCase):
def test_multi_objective_reward(self):
from skillopt.sleep.replay import multi_objective_reward
from skillopt.sleep.types import ReplayResult, TaskRecord
from skillopt_sleep.replay import multi_objective_reward
from skillopt_sleep.types import ReplayResult, TaskRecord
t = TaskRecord(id="t", project="/p", intent="x")
expensive = [(t, ReplayResult(id="t", hard=1.0, tokens=4000, latency_ms=20000))]
cheap = [(t, ReplayResult(id="t", hard=1.0, tokens=200, latency_ms=1000))]
@@ -248,8 +248,8 @@ class TestMultiObjectiveAndPrefs(unittest.TestCase):
self.assertGreater(rc, re)
def test_preferences_injected_into_reflect(self):
from skillopt.sleep.backend import CliBackend
from skillopt.sleep.types import TaskRecord, ReplayResult
from skillopt_sleep.backend import CliBackend
from skillopt_sleep.types import TaskRecord, ReplayResult
captured = {}
class CapBackend(CliBackend):
@@ -267,9 +267,9 @@ class TestMultiObjectiveAndPrefs(unittest.TestCase):
self.assertIn("British English", captured["prompt"])
def test_replay_records_cost(self):
from skillopt.sleep.backend import MockBackend
from skillopt.sleep.replay import replay_one
from skillopt.sleep.types import TaskRecord
from skillopt_sleep.backend import MockBackend
from skillopt_sleep.replay import replay_one
from skillopt_sleep.types import TaskRecord
t = TaskRecord(id="t", project="/p", intent="hello world",
reference_kind="exact", reference="hi")
r = replay_one(MockBackend(), t, "some skill text", "")
@@ -279,8 +279,8 @@ class TestMultiObjectiveAndPrefs(unittest.TestCase):
class TestMultiRolloutAndBudget(unittest.TestCase):
def test_rolloutset_stats(self):
from skillopt.sleep.rollout import RolloutSet
from skillopt.sleep.types import ReplayResult, TaskRecord
from skillopt_sleep.rollout import RolloutSet
from skillopt_sleep.types import ReplayResult, TaskRecord
rs = RolloutSet(task=TaskRecord(id="t", project="/p", intent="x"),
attempts=[ReplayResult(id="t", hard=1.0),
ReplayResult(id="t", hard=0.0),
@@ -291,7 +291,7 @@ class TestMultiRolloutAndBudget(unittest.TestCase):
self.assertAlmostEqual(rs.pass_rate, 2 / 3)
def test_budget_exhaustion_and_plan(self):
from skillopt.sleep.budget import Budget, plan_depth
from skillopt_sleep.budget import Budget, plan_depth
clock = [0.0]
b = Budget(max_tokens=1000)
b.start(lambda: clock[0], tokens_now=0)
@@ -303,9 +303,9 @@ class TestMultiRolloutAndBudget(unittest.TestCase):
self.assertGreaterEqual(k, 1)
def test_contrastive_reflect_with_stub(self):
from skillopt.sleep.backend import Backend
from skillopt.sleep.rollout import RolloutSet, contrastive_reflect
from skillopt.sleep.types import ReplayResult, TaskRecord
from skillopt_sleep.backend import Backend
from skillopt_sleep.rollout import RolloutSet, contrastive_reflect
from skillopt_sleep.types import ReplayResult, TaskRecord
class StubBackend(Backend):
name = "stub"
@@ -322,7 +322,7 @@ class TestMultiRolloutAndBudget(unittest.TestCase):
class TestSlowUpdate(unittest.TestCase):
def test_protected_field_roundtrip(self):
from skillopt.sleep.slow_update import (
from skillopt_sleep.slow_update import (
replace_slow_field, extract_slow_field, has_slow_field,
SLOW_UPDATE_START, SLOW_UPDATE_END,
)
@@ -339,9 +339,9 @@ class TestSlowUpdate(unittest.TestCase):
self.assertIn("keep me", doc2)
def test_run_slow_update_with_stub_backend(self):
from skillopt.sleep.backend import Backend
from skillopt.sleep.slow_update import run_slow_update
from skillopt.sleep.types import TaskRecord, ReplayResult
from skillopt_sleep.backend import Backend
from skillopt_sleep.slow_update import run_slow_update
from skillopt_sleep.types import TaskRecord, ReplayResult
class StubBackend(Backend):
name = "stub"
@@ -365,10 +365,10 @@ class TestSlowUpdate(unittest.TestCase):
class TestToolLoop(unittest.TestCase):
def test_tool_called_judge_via_replay(self):
from skillopt.sleep.backend import MockBackend
from skillopt.sleep.replay import replay_one, _required_tools
from skillopt.sleep.memory import set_learned
from skillopt.sleep.types import TaskRecord
from skillopt_sleep.backend import MockBackend
from skillopt_sleep.replay import replay_one, _required_tools
from skillopt_sleep.memory import set_learned
from skillopt_sleep.types import TaskRecord
task = TaskRecord(
id="qa1", project="/p", intent="answer the question",