github/microsoft-SkillOpt
1
0
Fork 0
mirror of https://github.com/microsoft/SkillOpt.git synced 2026-08-03 07:02:46 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(sleep): harden Copilot transcript harvesting

Browse Source 
Signed-off-by: Yifan Yang <29210256+Yif-Yang@users.noreply.github.com>
This commit is contained in:
Yifan Yang
2026-08-01 16:59:08 +00:00
parent 548af6576b
commit 925d76222a
5 changed files with 238 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
docs/reference/cli.md
View File

@@ -4,8 +4,8 @@
> include the generic research `openai_compatible` backend, Sleep handoff,
> Sleep support for non-Azure OpenAI-compatible endpoints, the Sleep
> `--preferences` flag, the research `cursor_exec` target harness, or Cursor
> source/backend/plugin support; use a source install from `main` for those
> features until the next release.
> source/backend/plugin support or VS Code Copilot transcript harvesting; use
> a source install from `main` for those features until the next release.
## Training
@@ -128,10 +128,11 @@ Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`, and
|---|---|
| `--project PATH` | Project used for transcript scope, targets, state, and staging (default: current directory) |
| `--scope invoked\|all` | Harvest this project or all projects |
| `--source claude\|codex\|cursor\|auto` | Transcript source; `auto` keeps Codex-then-Claude precedence and does not select Cursor |
| `--source claude\|codex\|copilot\|cursor\|auto` | Transcript source; `auto` keeps Codex-then-Claude precedence and does not select Copilot or Cursor |
| `--backend mock\|claude\|codex\|copilot\|cursor\|handoff\|azure_openai` | Replay/optimizer backend |
| `--model NAME` | Backend-specific model override |
| `--cursor-home PATH` | Override `~/.cursor` for Cursor transcript harvesting |
| `--vscode-workspace-storage PATH` | Override VS Code's `User/workspaceStorage` root for Copilot transcript harvesting |
| `--cursor-path PATH` | Path to the installed Cursor Agent CLI |
| `--preferences TEXT` | House rules supplied to reflection |
| `--lookback-hours N` | Initial transcript lookback; `0` scans all history |
@@ -142,6 +143,30 @@ Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`, and
| `--progress` / `--json` | Progress or machine-readable output |
| `--auto-adopt` | Apply an accepted staged proposal automatically |
### VS Code GitHub Copilot Chat source
`--source copilot` reads local VS Code GitHub Copilot Chat session logs from
the platform's stable and Insiders `User/workspaceStorage` locations, plus the
portable location when `VSCODE_PORTABLE` is available. Use
`--vscode-workspace-storage PATH` for a nonstandard root. Each workspace must
have a readable adjacent `workspace.json` mapping to a local project; unmapped
windows are skipped so project scoping cannot silently mix unrelated sessions.
`--source auto` does not select Copilot.
The harvester retains user-entered prompts, visible assistant Markdown, and
tool names from requests confirmed as GitHub Copilot Chat. It excludes
system-initiated notifications, reasoning, tool inputs and outputs, rendered
context, and account/model metadata. Known secret-shaped strings are redacted
as defense in depth, but review harvested tasks before sending them to a model
provider. Transcript harvesting is independent of `--backend copilot`, which
invokes the GitHub Copilot CLI for model calls.
The managed `schedule` command does not persist `--source` or
`--vscode-workspace-storage`. Before scheduling this source, set
`"transcript_source": "copilot"` and, when needed,
`"vscode_workspace_storage": "/absolute/path/to/workspaceStorage"` in
`~/.skillopt-sleep/config.json`.
### Cursor source and backend
`--source cursor` reads local Cursor JSONL transcripts from

41
docs/sleep/README.md
View File

@@ -17,7 +17,7 @@ normal agent requests.
One "night":
```
harvest Claude Code / Codex / Cursor transcripts → mine recurring tasks → replay via the configured backend (isolation varies by backend; mock/handoff make no network calls)
harvest Claude Code / Codex / VS Code Copilot / Cursor transcripts → mine recurring tasks → replay via the configured backend (isolation varies by backend; mock/handoff make no network calls)
→ consolidate (reflect → bounded edit → GATE on real held-out tasks)
→ stage proposal → (you) adopt
```
@@ -43,6 +43,12 @@ experience → long-term competence).
> Cursor and the model provider selected by Cursor may therefore receive
> transcript-derived content.
>
> The VS Code Copilot source reads local user-entered prompts, visible assistant
> Markdown, and tool names from confirmed GitHub Copilot Chat requests. It
> excludes system notifications, reasoning, tool inputs/outputs, rendered
> context, and account/model metadata. Known secret-shaped strings are
> redacted, but this remains defense in depth rather than a guarantee.
>
> By default, each stateful night also writes a local `evidence.jsonl` under
> the project staging tree (beside the report when one is staged); dry-runs
> write evidence under the configured Sleep state directory. The log contains
@@ -66,9 +72,10 @@ skillopt-sleep schedule # install a nightly cron entry for this project
```
> **Version note.** This page tracks `main`. PyPI 0.2.0 provides the base
> commands above. Cursor source/backend/plugin support, Sleep handoff, non-Azure
> OpenAI-compatible endpoints, and `--preferences` landed later and require a
> source install from `main` until the next release.
> commands above. Cursor source/backend/plugin support, VS Code Copilot
> transcript harvesting, Sleep handoff, non-Azure OpenAI-compatible endpoints,
> and `--preferences` landed later and require a source install from `main`
> until the next release.
The per-agent integrations below still come from the repo; the CLI above is the
standalone, pip-only way to run a cycle. Claude Code, Codex, Cursor, Copilot, and
@@ -86,6 +93,32 @@ One engine, thin per-agent shells (see [`plugins/`](https://github.com/microsoft
| **Devin** | [`plugins/devin`](https://github.com/microsoft/SkillOpt/tree/main/plugins/devin) | register `plugins/devin/mcp_server.py` as an MCP server |
| **OpenClaw** | [`plugins/openclaw`](https://github.com/microsoft/SkillOpt/tree/main/plugins/openclaw) | adapt the reference wrapper and paths for your installation |
### VS Code GitHub Copilot Chat
Use `--source copilot` to harvest local VS Code GitHub Copilot Chat sessions.
SkillOpt auto-detects stable and Insiders VS Code `User/workspaceStorage`
roots, plus the portable root when `VSCODE_PORTABLE` is available. Override
discovery with `--vscode-workspace-storage PATH`. Project scoping comes from
each storage entry's adjacent `workspace.json`, and entries without a safe
local mapping are skipped. `--source auto` retains Codex-then-Claude precedence
and does not select Copilot.
```bash
skillopt-sleep harvest --project "$(pwd)" --source copilot --progress
skillopt-sleep dry-run --project "$(pwd)" --source copilot --backend copilot
```
The source and backend are independent: `--source copilot` reads VS Code's
local history, while `--backend copilot` uses the separately installed and
authenticated GitHub Copilot CLI for mining, replay, judging, and reflection.
Inspect harvested tasks before using a real backend on sensitive projects.
The managed scheduler does not preserve `--source` or
`--vscode-workspace-storage`. Before scheduling this source, put
`"transcript_source": "copilot"` and, for a nonstandard root,
`"vscode_workspace_storage": "/absolute/path/to/workspaceStorage"` in
`~/.skillopt-sleep/config.json`.
### Cursor
Cursor transcript harvesting and model execution are independent. Use

1
skillopt_sleep/__main__.py
View File

@@ -15,6 +15,7 @@ Common flags:
--tasks-file PATH reviewed TaskRecord JSON file to replay instead of harvesting
--backend mock|claude|codex|copilot|cursor|handoff
--source claude|codex|copilot|cursor|auto
--vscode-workspace-storage PATH
--model NAME
--lookback-hours N
--auto-adopt

45
skillopt_sleep/harvest_copilot.py
View File

@@ -47,9 +47,9 @@ def default_vscode_workspace_storage_roots(
env: Optional[Mapping[str, str]] = None,
home: Optional[str] = None,
) -> List[str]:
"""Return stable and Insiders ``workspaceStorage`` candidates for an OS."""
"""Return stable, Insiders, and portable ``workspaceStorage`` candidates."""
platform = platform or sys.platform
env = env or os.environ
env = os.environ if env is None else env
home = os.path.expanduser(home or "~")
if platform == "win32":
@@ -104,7 +104,11 @@ def _workspace_project(workspace_dir: str) -> str:
def discover_vscode_sessions(workspace_storage: str = "") -> List[CopilotSessionFile]:
"""Discover VS Code chat JSONL files, newest first, without opening session content."""
"""Discover chat logs newest first and read adjacent project mappings.
Session content remains unopened during discovery; only the neighboring
``workspace.json`` is read to establish project scope.
"""
roots = [os.path.abspath(os.path.expanduser(workspace_storage))] if workspace_storage else (
default_vscode_workspace_storage_roots()
)
@@ -165,6 +169,7 @@ def _read_jsonl_snapshot(path: str) -> Iterable[Dict[str, Any]]:
except (FileNotFoundError, IsADirectoryError, PermissionError, OSError):
return
def _resolve_parent(root: Any, path: Any) -> tuple[Any, Any] | tuple[None, None]:
if not isinstance(path, list) or not path:
return None, None
@@ -319,6 +324,21 @@ def _iso_timestamp(value: Any) -> str:
return ""
def _iso_epoch(value: Optional[str]) -> Optional[float]:
"""Parse an ISO-8601 timestamp for chronological comparisons."""
if not value:
return None
try:
normalized = value[:-1] + "+00:00" if value[-1:] in {"Z", "z"} else value
parsed = datetime.fromisoformat(normalized)
# Sleep state checkpoints are local-time strings without an offset.
# ``timestamp()`` intentionally interprets those in the host timezone,
# matching the existing Cursor harvester and the producer in state.py.
return parsed.timestamp()
except (OverflowError, OSError, TypeError, ValueError):
return None
def digest_copilot_session(path: str, project: str = "") -> Optional[SessionDigest]:
"""Build a privacy-bounded digest from one reconstructed VS Code session."""
session = reconstruct_chat_session(path)
@@ -339,7 +359,11 @@ def digest_copilot_session(path: str, project: str = "") -> Optional[SessionDige
if not isinstance(requests, list):
requests = []
for request in requests:
if not isinstance(request, dict) or _extension_id(request) != _COPILOT_EXTENSION_ID:
if (
not isinstance(request, dict)
or _extension_id(request) != _COPILOT_EXTENSION_ID
or request.get("isSystemInitiated") is True
):
continue
message = request.get("message")
prompt = _sanitize_text(message.get("text") if isinstance(message, dict) else "")
@@ -393,6 +417,7 @@ def harvest_copilot(
) -> List[SessionDigest]:
"""Discover and normalize matching VS Code GitHub Copilot Chat sessions."""
digests: List[SessionDigest] = []
since_epoch = _iso_epoch(since_iso)
for session_file in discover_vscode_sessions(workspace_storage):
digest = digest_copilot_session(session_file.path, project=session_file.project)
if digest is None or _is_headless_replay(digest) or _is_agent_session(digest):
@@ -402,8 +427,16 @@ def harvest_copilot(
continue
if not _project_matches(digest.project or "", scope, invoked_project):
continue
if since_iso and digest.ended_at and digest.ended_at < since_iso:
continue
if since_iso and digest.ended_at:
ended_epoch = _iso_epoch(digest.ended_at)
if since_epoch is not None and ended_epoch is not None:
before_cutoff = ended_epoch < since_epoch
else:
# Preserve the previous best-effort behavior for malformed or
# legacy timestamps that ``datetime.fromisoformat`` cannot parse.
before_cutoff = digest.ended_at < since_iso
if before_cutoff:
continue
digests.append(digest)
if limit and len(digests) >= limit:
break

133
tests/test_copilot_harvest.py
View File

@@ -6,7 +6,9 @@ import json
import os
import shutil
import tempfile
import time
import unittest
from datetime import datetime
from unittest import mock
from skillopt_sleep.config import load_config
@@ -16,6 +18,7 @@ from skillopt_sleep.harvest_copilot import (
discover_vscode_sessions,
harvest_copilot,
reconstruct_chat_session,
_iso_epoch,
)
from skillopt_sleep.harvest_sources import harvest_for_config
@@ -118,6 +121,43 @@ class TestCopilotHarvest(unittest.TestCase):
self.assertIsNone(digest)
def test_digest_skips_system_initiated_requests(self):
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "system-request.jsonl")
snapshot = {
"kind": 0,
"v": {
"version": 3,
"sessionId": "system-request",
"requests": [
{
"timestamp": 1700000001000,
"agent": {"extensionId": {"_lower": "github.copilot-chat"}},
"message": {"text": "real user prompt"},
"response": [{"value": "visible answer"}],
},
{
"timestamp": 1700000009000,
"isSystemInitiated": True,
"agent": {"extensionId": {"_lower": "github.copilot-chat"}},
"message": {"text": "terminal command completed"},
"response": [{"value": "system notification response"}],
},
],
},
}
with open(path, "w", encoding="utf-8") as f:
f.write(json.dumps(snapshot) + "\n")
digest = digest_copilot_session(path, project="/tmp/copilot-project")
self.assertIsNotNone(digest)
self.assertEqual(digest.user_prompts, ["real user prompt"])
self.assertEqual(digest.assistant_finals, ["visible answer"])
self.assertEqual(digest.ended_at, "2023-11-14T22:13:21Z")
self.assertEqual(digest.n_user_turns, 1)
self.assertEqual(digest.n_assistant_turns, 1)
def test_discovers_sessions_and_resolves_workspace_folder(self):
with tempfile.TemporaryDirectory() as tmp:
_workspace, path = self._storage_fixture(tmp)
@@ -155,6 +195,50 @@ class TestCopilotHarvest(unittest.TestCase):
self.assertEqual(digests[0].project, os.path.abspath("/tmp/copilot-project"))
self.assertEqual(old_digests, [])
def test_harvest_compares_fractional_timestamps_chronologically(self):
with tempfile.TemporaryDirectory() as tmp:
_workspace, path = self._storage_fixture(tmp)
with open(path, "a", encoding="utf-8") as f:
f.write(json.dumps({
"kind": 1,
"k": ["requests", 0, "modelState", "completedAt"],
"v": 1700000010500,
}) + "\n")
included = harvest_copilot(
tmp, scope="all", since_iso="2023-11-14T22:13:30Z"
)
excluded = harvest_copilot(
tmp, scope="all", since_iso="2023-11-14T22:13:30.750000Z"
)
offset_included = harvest_copilot(
tmp, scope="all", since_iso="2023-11-14T17:13:30-05:00"
)
self.assertEqual(len(included), 1)
self.assertEqual(included[0].ended_at, "2023-11-14T22:13:30.500000Z")
self.assertEqual(excluded, [])
self.assertEqual(len(offset_included), 1)
@unittest.skipUnless(hasattr(time, "tzset"), "requires POSIX timezone control")
def test_naive_checkpoint_uses_local_timezone_like_sleep_state(self):
previous_tz = os.environ.get("TZ")
try:
# Force a non-UTC local zone so treating the naive value as UTC
# would make this assertion fail.
os.environ["TZ"] = "UTC-5"
time.tzset()
cutoff = 1_700_000_000
local_checkpoint = datetime.fromtimestamp(cutoff).isoformat()
self.assertEqual(_iso_epoch(local_checkpoint), cutoff)
finally:
if previous_tz is None:
os.environ.pop("TZ", None)
else:
os.environ["TZ"] = previous_tz
time.tzset()
def test_missing_storage_returns_empty_result(self):
self.assertEqual(discover_vscode_sessions("/definitely/missing/workspaceStorage"), [])
self.assertEqual(harvest_copilot("/definitely/missing/workspaceStorage"), [])
@@ -190,6 +274,35 @@ class TestCopilotHarvest(unittest.TestCase):
self.assertTrue(linux[0].startswith("/config"))
self.assertIn("AppData", windows[0])
def test_explicit_empty_environment_remains_isolated(self):
with mock.patch.dict(
os.environ,
{
"XDG_CONFIG_HOME": "/host-config",
"VSCODE_PORTABLE": "/host-portable",
},
clear=True,
):
candidates_by_platform = {
platform: default_vscode_workspace_storage_roots(
platform=platform, env={}, home=home
)
for platform, home in (
("linux", "/home/test"),
("darwin", "/Users/test"),
("win32", "C:/Users/test"),
)
}
for candidates in candidates_by_platform.values():
self.assertEqual(len(candidates), 2)
self.assertNotIn("/host-config", "\n".join(candidates))
self.assertNotIn("/host-portable", "\n".join(candidates))
self.assertEqual(
candidates_by_platform["linux"][0],
os.path.join("/home/test", ".config", "Code", "User", "workspaceStorage"),
)
def test_portable_storage_uses_vscode_portable_data_root(self):
candidates = default_vscode_workspace_storage_roots(
platform="linux",
@@ -222,6 +335,26 @@ class TestCopilotHarvest(unittest.TestCase):
self.assertEqual(len(digests), 1)
self.assertEqual(digests[0].session_id, "fixture-session")
def test_documented_user_config_keys_route_copilot_source(self):
with tempfile.TemporaryDirectory() as tmp:
self._storage_fixture(tmp)
config_path = os.path.join(tmp, "config.json")
with open(config_path, "w", encoding="utf-8") as f:
json.dump({
"transcript_source": "copilot",
"vscode_workspace_storage": tmp,
"projects": "invoked",
}, f)
with mock.patch(
"skillopt_sleep.config._user_config_path", return_value=config_path
):
cfg = load_config(invoked_project="/tmp/copilot-project")
digests = harvest_for_config(cfg, limit=10)
self.assertEqual(cfg.get("transcript_source"), "copilot")
self.assertEqual(cfg.vscode_workspace_storage, os.path.abspath(tmp))
self.assertEqual(len(digests), 1)
def test_auto_source_behavior_remains_codex_then_claude(self):
cfg = load_config(
transcript_source="auto",