mirror of
https://github.com/microsoft/SkillOpt.git
synced 2026-08-03 07:02:46 +08:00
Merge upstream/main into feat/cursor-skillopt-sleep
This commit is contained in:
@@ -11,6 +11,10 @@ All notable changes to SkillOpt are documented here. This project adheres to
|
||||
and skill, Cursor transcript harvesting, and an optional Cursor Agent CLI
|
||||
backend. Cursor tool-aware replay remains disabled pending live permission-
|
||||
boundary validation.
|
||||
- **Cursor Agent research target harness** (`cursor_exec`) for running
|
||||
supported benchmark rollouts through an installed, authenticated
|
||||
`cursor-agent`, with sandboxed workspaces, structured trace capture, and
|
||||
target-only optimizer separation.
|
||||
- **Handoff backend** (`--backend handoff`) for SkillOpt-Sleep — runs the
|
||||
sleep cycle with no model subprocess or API key: the engine writes each
|
||||
pending model call to `PROMPTS.md`/`pending.json` (exit code 3) and the
|
||||
|
||||
10
README.md
10
README.md
@@ -65,13 +65,13 @@ https://github.com/user-attachments/assets/eb12d3bc-371c-467f-904d-91b61f339ed7
|
||||
|
||||
A backend = a chat / exec target (e.g. `openai_chat`, `claude_chat`,
|
||||
`qwen_chat`, `minimax_chat`, `openai_compatible`, `codex_exec`,
|
||||
`claude_code_exec`). If a provider implements the OpenAI Chat Completions
|
||||
`claude_code_exec`, `cursor_exec`). If a provider implements the OpenAI Chat Completions
|
||||
protocol, try the built-in `openai_compatible` backend before adding code. See
|
||||
[`docs/guide/new-backend.md`](docs/guide/new-backend.md) for the full
|
||||
contract; in short you add a `skillopt/model/<name>_backend.py` module,
|
||||
register it in `skillopt/model/common.py` + `backend_config.py`, and wire
|
||||
it through the router in `skillopt/model/__init__.py`. `qwen_backend.py`
|
||||
and `minimax_backend.py` are good templates.
|
||||
contract. Chat backends add a `skillopt/model/<name>_backend.py` module;
|
||||
target-only exec backends use the shared harness in `codex_harness.py`.
|
||||
Both register through `common.py`, `backend_config.py`, and
|
||||
`skillopt/model/__init__.py`.
|
||||
|
||||
### Adding a new benchmark
|
||||
|
||||
|
||||
@@ -24,6 +24,8 @@ model:
|
||||
claude_code_exec_use_sdk: auto
|
||||
claude_code_exec_effort: medium
|
||||
claude_code_exec_max_thinking_tokens: 16384
|
||||
cursor_exec_path: "" # blank uses CURSOR_EXEC_PATH or cursor-agent
|
||||
cursor_exec_sandbox: "" # blank uses CURSOR_EXEC_SANDBOX or enabled
|
||||
codex_trace_to_optimizer: true
|
||||
azure_openai_endpoint: "" # e.g. "https://your-resource.openai.azure.com/"
|
||||
azure_openai_api_version: "2024-12-01-preview"
|
||||
|
||||
@@ -45,6 +45,7 @@ model:
|
||||
| `minimax_chat` | ✓ | ✓ | MiniMax API |
|
||||
| `codex_exec` | — | ✓ | Codex CLI execution harness |
|
||||
| `claude_code_exec` | — | ✓ | Claude Code CLI execution harness |
|
||||
| `cursor_exec` | — | ✓ | Cursor Agent CLI execution harness |
|
||||
|
||||
The current MiniMax adapter has one shared deployment. Set
|
||||
`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot
|
||||
@@ -183,6 +184,9 @@ Model credentials are loaded from environment variables:
|
||||
| `OPENAI_COMPATIBLE_MODEL` | `openai_compatible` | Shared provider model ID for direct library use; train/eval YAML role models take precedence |
|
||||
| `CLAUDE_CLI_BIN` | `claude_chat` | Optional path to the `claude` executable; defaults to `claude` |
|
||||
| `ANTHROPIC_API_KEY` | `claude_chat` | Optional authentication method understood by the Claude CLI, not a direct SkillOpt API client |
|
||||
| `CURSOR_EXEC_PATH` | `cursor_exec` | Optional path to `cursor-agent`; defaults to `cursor-agent` |
|
||||
| `CURSOR_EXEC_SANDBOX` | `cursor_exec` | Cursor sandbox mode: `enabled` (default) or `disabled` |
|
||||
| `CURSOR_API_KEY` | `cursor_exec` | Optional authentication method understood directly by Cursor Agent |
|
||||
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | Local Qwen/vLLM endpoint |
|
||||
| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence |
|
||||
| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL |
|
||||
@@ -197,6 +201,14 @@ and authenticate that CLI before use. Setting `ANTHROPIC_API_KEY` is one way
|
||||
the CLI may authenticate, but SkillOpt does not call the Anthropic API
|
||||
directly through this backend.
|
||||
|
||||
`cursor_exec` is a target-only benchmark harness. Install and authenticate
|
||||
Cursor Agent first, then select it with `model.target_backend=cursor_exec`.
|
||||
Read-only rollouts use Ask mode; artifact-producing rollouts add Cursor's
|
||||
headless `--force` flag inside the benchmark workspace. SkillOpt enables the
|
||||
Cursor sandbox by default and rejects file-edit rollouts if it is disabled;
|
||||
read-only Ask-mode rollouts may explicitly disable it. SkillOpt does not approve
|
||||
MCP servers automatically.
|
||||
|
||||
### Three OpenAI-compatible paths
|
||||
|
||||
- Research, generic provider: select `openai_compatible` and use
|
||||
|
||||
@@ -146,9 +146,9 @@ Provider-specific configuration helpers and `count_tokens()` are optional, but
|
||||
their state must be safe to update while calls may run concurrently. Keep
|
||||
credentials out of logs and persisted artifacts.
|
||||
|
||||
Exec-style targets do not implement this chat contract. They are target-only
|
||||
and are integrated through `codex_harness.py` plus environment-specific rollout
|
||||
code.
|
||||
Exec-style targets such as `claude_code_exec` and `cursor_exec` do not
|
||||
implement this chat contract. They are target-only and are integrated through
|
||||
`codex_harness.py` plus environment-specific rollout code.
|
||||
|
||||
## Step 2: register and route the backend
|
||||
|
||||
|
||||
@@ -397,6 +397,7 @@ python scripts/train.py --config configs/searchqa/default.yaml</code></pre>
|
||||
<tr><td><code>minimax_chat</code></td><td>Yes</td><td>Yes</td><td>MiniMax chat endpoint.</td></tr>
|
||||
<tr><td><code>codex_exec</code></td><td>Yes</td><td>Supported adapters only</td><td>Executes Codex for optimizer calls and as a target agent where supported.</td></tr>
|
||||
<tr><td><code>claude_code_exec</code></td><td>No</td><td>Supported adapters only</td><td>Executes Claude Code as a target agent.</td></tr>
|
||||
<tr><td><code>cursor_exec</code></td><td>No</td><td>Supported adapters only</td><td>Executes Cursor Agent as a sandboxed target agent where supported.</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
@@ -192,6 +192,7 @@ not via a base class subclass. Supported values (as of this writing):
|
||||
| `openai_compatible` | ✓ | ✓ |
|
||||
| `codex_exec` | ✓ | ✓ |
|
||||
| `claude_code_exec` | — | ✓ |
|
||||
| `cursor_exec` | — | ✓ |
|
||||
|
||||
See `skillopt/model/backend_config.py` for the live whitelist and
|
||||
[`docs/reference/config.md`](./config.md) for the per-backend
|
||||
|
||||
@@ -3,8 +3,9 @@
|
||||
> **Version note.** This reference tracks `main`. PyPI 0.2.0 does not yet
|
||||
> include the generic research `openai_compatible` backend, Sleep handoff,
|
||||
> Sleep support for non-Azure OpenAI-compatible endpoints, the Sleep
|
||||
> `--preferences` flag, or Cursor source/backend/plugin support; use a source
|
||||
> install from `main` for those features until the next release.
|
||||
> `--preferences` flag, the research `cursor_exec` target harness, or Cursor
|
||||
> source/backend/plugin support; use a source install from `main` for those
|
||||
> features until the next release.
|
||||
|
||||
## Training
|
||||
|
||||
@@ -91,6 +92,27 @@ python scripts/train.py \
|
||||
model.target=deepseek-chat
|
||||
```
|
||||
|
||||
To benchmark an installed, authenticated Cursor Agent through an environment
|
||||
that supports exec targets:
|
||||
|
||||
```bash
|
||||
python scripts/eval_only.py \
|
||||
--config configs/searchqa/default.yaml \
|
||||
--skill skills/my_skill.md \
|
||||
--cfg-options \
|
||||
model.optimizer_backend=openai_chat \
|
||||
model.target_backend=cursor_exec \
|
||||
model.target=composer-2.5
|
||||
```
|
||||
|
||||
`cursor_exec` runs the target only; the optimizer remains separately
|
||||
configured. Read-only rollouts use Cursor Ask mode. Rollouts that request file
|
||||
edits use `--force` inside the benchmark workspace, with Cursor sandboxing
|
||||
enabled. The harness refuses file-edit rollouts when the Cursor sandbox is
|
||||
disabled. Read-only Ask-mode rollouts may explicitly disable it. Override the
|
||||
executable or sandbox through `model.cursor_exec_path` and
|
||||
`model.cursor_exec_sandbox`.
|
||||
|
||||
## SkillOpt-Sleep
|
||||
|
||||
```bash
|
||||
|
||||
@@ -18,6 +18,7 @@ selecting the generic OpenAI-compatible backend.
|
||||
| `minimax_chat` | ✓ | ✓ |
|
||||
| `codex_exec` | ✓ | ✓ |
|
||||
| `claude_code_exec` | — | ✓ |
|
||||
| `cursor_exec` | — | ✓ |
|
||||
|
||||
MiniMax currently has one shared deployment. `model.minimax_model` is applied
|
||||
when MiniMax is the target; mixed-backend runs cannot independently choose a
|
||||
@@ -64,6 +65,8 @@ defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`.
|
||||
| `model.minimax_*` | MiniMax `base_url`, `api_key`, shared `minimax_model`, `temperature`, `max_tokens`, and `enable_thinking`; `minimax_model` applies when MiniMax is the target |
|
||||
| `model.codex_exec_*` | Codex path, sandbox, profile, SDK mode, reasoning, network/search, and approval policy |
|
||||
| `model.claude_code_exec_*` | Claude path, profile, SDK mode, effort, and thinking-token cap |
|
||||
| `model.cursor_exec_path` | Cursor Agent executable path; default `cursor-agent` |
|
||||
| `model.cursor_exec_sandbox` | Cursor sandbox mode: `enabled` (default) or `disabled`; file-edit rollouts require `enabled` |
|
||||
|
||||
## Training (`train`)
|
||||
|
||||
|
||||
@@ -28,6 +28,7 @@ from skillopt.model import (
|
||||
configure_azure_openai,
|
||||
configure_claude_code_exec,
|
||||
configure_codex_exec,
|
||||
configure_cursor_exec,
|
||||
configure_qwen_chat,
|
||||
configure_minimax_chat,
|
||||
set_reasoning_effort,
|
||||
@@ -139,7 +140,7 @@ def parse_args() -> argparse.Namespace:
|
||||
# Legacy flat overrides
|
||||
p.add_argument("--env", type=str)
|
||||
p.add_argument("--backend", type=str,
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "minimax", "minimax_chat"])
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "cursor", "cursor_exec", "minimax", "minimax_chat"])
|
||||
p.add_argument("--optimizer_model", type=str)
|
||||
p.add_argument("--target_model", type=str)
|
||||
p.add_argument("--optimizer_backend", type=str)
|
||||
@@ -181,6 +182,8 @@ def parse_args() -> argparse.Namespace:
|
||||
p.add_argument("--claude_code_exec_use_sdk", type=str)
|
||||
p.add_argument("--claude_code_exec_effort", type=str)
|
||||
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
|
||||
p.add_argument("--cursor_exec_path", type=str)
|
||||
p.add_argument("--cursor_exec_sandbox", type=str)
|
||||
p.add_argument("--minimax_base_url", type=str)
|
||||
p.add_argument("--minimax_api_key", type=str)
|
||||
p.add_argument("--minimax_model", type=str)
|
||||
@@ -262,6 +265,8 @@ def main() -> None:
|
||||
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
|
||||
"claude_code_exec_effort": "model.claude_code_exec_effort",
|
||||
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
|
||||
"cursor_exec_path": "model.cursor_exec_path",
|
||||
"cursor_exec_sandbox": "model.cursor_exec_sandbox",
|
||||
"minimax_base_url": "model.minimax_base_url",
|
||||
"minimax_api_key": "model.minimax_api_key",
|
||||
"minimax_model": "model.minimax_model",
|
||||
@@ -327,6 +332,11 @@ def main() -> None:
|
||||
elif backend == "claude_code_exec":
|
||||
cfg.setdefault("optimizer_backend", "openai_chat")
|
||||
cfg.setdefault("target_backend", "claude_code_exec")
|
||||
elif backend == "cursor_exec":
|
||||
if not _has_model_override("model.optimizer_backend", "optimizer_backend"):
|
||||
cfg["optimizer_backend"] = "openai_chat"
|
||||
if not _has_model_override("model.target_backend", "target_backend"):
|
||||
cfg["target_backend"] = "cursor_exec"
|
||||
elif backend in {"minimax", "minimax_chat"}:
|
||||
cfg.setdefault("optimizer_backend", "openai_chat")
|
||||
cfg.setdefault("target_backend", "minimax_chat")
|
||||
@@ -355,6 +365,12 @@ def main() -> None:
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
cfg["target_model"] = default_model_for_backend("claude_chat")
|
||||
if cfg.get("target_backend") == "cursor_exec":
|
||||
if (
|
||||
str(cfg.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
cfg["target_model"] = default_model_for_backend("cursor_exec")
|
||||
if cfg.get("target_backend") == "minimax_chat":
|
||||
if (
|
||||
str(cfg.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
@@ -429,6 +445,10 @@ def main() -> None:
|
||||
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
|
||||
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
|
||||
)
|
||||
configure_cursor_exec(
|
||||
path=cfg.get("cursor_exec_path") or None,
|
||||
sandbox=cfg.get("cursor_exec_sandbox") or None,
|
||||
)
|
||||
configure_qwen_chat(
|
||||
base_url=cfg.get("qwen_chat_base_url") or None,
|
||||
api_key=cfg.get("qwen_chat_api_key") or None,
|
||||
|
||||
@@ -137,7 +137,7 @@ def parse_args() -> argparse.Namespace:
|
||||
# Legacy flat CLI overrides (still work, prefer --cfg-options for new usage)
|
||||
p.add_argument("--env", type=str)
|
||||
p.add_argument("--backend", type=str,
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
|
||||
choices=["azure_openai", "codex", "codex_exec", "claude", "claude_chat", "claude_code_exec", "cursor", "cursor_exec", "qwen", "qwen_chat", "minimax", "minimax_chat"])
|
||||
p.add_argument("--optimizer_model", type=str)
|
||||
p.add_argument("--target_model", type=str)
|
||||
p.add_argument("--optimizer_backend", type=str)
|
||||
@@ -205,6 +205,8 @@ def parse_args() -> argparse.Namespace:
|
||||
p.add_argument("--claude_code_exec_use_sdk", type=str)
|
||||
p.add_argument("--claude_code_exec_effort", type=str)
|
||||
p.add_argument("--claude_code_exec_max_thinking_tokens", type=int)
|
||||
p.add_argument("--cursor_exec_path", type=str)
|
||||
p.add_argument("--cursor_exec_sandbox", type=str)
|
||||
p.add_argument("--codex_trace_to_optimizer", type=_BOOL)
|
||||
p.add_argument("--skill_init", type=str)
|
||||
p.add_argument("--num_epochs", type=int)
|
||||
@@ -343,6 +345,8 @@ _LEGACY_TO_STRUCTURED: dict[str, str] = {
|
||||
"claude_code_exec_use_sdk": "model.claude_code_exec_use_sdk",
|
||||
"claude_code_exec_effort": "model.claude_code_exec_effort",
|
||||
"claude_code_exec_max_thinking_tokens": "model.claude_code_exec_max_thinking_tokens",
|
||||
"cursor_exec_path": "model.cursor_exec_path",
|
||||
"cursor_exec_sandbox": "model.cursor_exec_sandbox",
|
||||
"codex_trace_to_optimizer": "model.codex_trace_to_optimizer",
|
||||
"num_epochs": "train.num_epochs",
|
||||
"train_size": "train.train_size",
|
||||
@@ -445,6 +449,11 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
elif backend == "claude_code_exec":
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "claude_code_exec")
|
||||
elif backend == "cursor_exec":
|
||||
if not _has_model_override("model.optimizer_backend", "optimizer_backend"):
|
||||
flat["optimizer_backend"] = "openai_chat"
|
||||
if not _has_model_override("model.target_backend", "target_backend"):
|
||||
flat["target_backend"] = "cursor_exec"
|
||||
elif backend in {"qwen", "qwen_chat"}:
|
||||
flat.setdefault("optimizer_backend", "openai_chat")
|
||||
flat.setdefault("target_backend", "qwen_chat")
|
||||
@@ -482,6 +491,12 @@ def load_config(args: argparse.Namespace) -> dict:
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("claude_chat")
|
||||
if flat.get("target_backend") == "cursor_exec":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
and not _has_model_override("model.target", "target_model")
|
||||
):
|
||||
flat["target_model"] = default_model_for_backend("cursor_exec")
|
||||
if flat.get("target_backend") == "qwen_chat":
|
||||
if (
|
||||
str(flat.get("target_model", "") or "").strip() in _OPENAI_DEFAULT_MODEL_SENTINELS
|
||||
|
||||
@@ -51,6 +51,8 @@ _FLATTEN_MAP: dict[str, str] = {
|
||||
"model.claude_code_exec_use_sdk": "claude_code_exec_use_sdk",
|
||||
"model.claude_code_exec_effort": "claude_code_exec_effort",
|
||||
"model.claude_code_exec_max_thinking_tokens": "claude_code_exec_max_thinking_tokens",
|
||||
"model.cursor_exec_path": "cursor_exec_path",
|
||||
"model.cursor_exec_sandbox": "cursor_exec_sandbox",
|
||||
"model.codex_trace_to_optimizer": "codex_trace_to_optimizer",
|
||||
"model.azure_endpoint": "azure_endpoint",
|
||||
"model.azure_api_version": "azure_api_version",
|
||||
|
||||
@@ -63,6 +63,7 @@ from skillopt.model import (
|
||||
configure_azure_openai,
|
||||
configure_claude_code_exec,
|
||||
configure_codex_exec,
|
||||
configure_cursor_exec,
|
||||
configure_minimax_chat,
|
||||
configure_qwen_chat,
|
||||
get_token_summary,
|
||||
@@ -681,6 +682,9 @@ class ReflACTTrainer:
|
||||
elif backend == "claude_code_exec":
|
||||
optimizer_backend = optimizer_backend or "openai_chat"
|
||||
target_backend = target_backend or "claude_code_exec"
|
||||
elif backend in {"cursor", "cursor_exec"}:
|
||||
optimizer_backend = optimizer_backend or "openai_chat"
|
||||
target_backend = target_backend or "cursor_exec"
|
||||
elif backend in {"qwen", "qwen_chat"}:
|
||||
optimizer_backend = optimizer_backend or "openai_chat"
|
||||
target_backend = target_backend or "qwen_chat"
|
||||
@@ -711,6 +715,10 @@ class ReflACTTrainer:
|
||||
effort=cfg.get("claude_code_exec_effort", cfg.get("reasoning_effort", "medium")),
|
||||
max_thinking_tokens=cfg.get("claude_code_exec_max_thinking_tokens", 16384),
|
||||
)
|
||||
configure_cursor_exec(
|
||||
path=cfg.get("cursor_exec_path") or None,
|
||||
sandbox=cfg.get("cursor_exec_sandbox") or None,
|
||||
)
|
||||
configure_qwen_chat(
|
||||
base_url=cfg.get("qwen_chat_base_url") or None,
|
||||
api_key=cfg.get("qwen_chat_api_key") or None,
|
||||
|
||||
@@ -51,5 +51,5 @@ evaluation:
|
||||
# Override only what differs from the inherited defaults.
|
||||
model:
|
||||
optimizer_backend: openai_chat # openai_chat | claude_chat | qwen_chat | minimax_chat | codex_exec
|
||||
target_backend: openai_chat # chat backends plus codex_exec / claude_code_exec
|
||||
target_backend: openai_chat # chat backends plus codex_exec / claude_code_exec / cursor_exec
|
||||
reasoning_effort: medium
|
||||
|
||||
@@ -13,8 +13,10 @@ from skillopt.model import qwen_backend as _qwen
|
||||
from skillopt.model.backend_config import ( # noqa: F401
|
||||
configure_claude_code_exec,
|
||||
configure_codex_exec,
|
||||
configure_cursor_exec,
|
||||
get_claude_code_exec_config,
|
||||
get_codex_exec_config,
|
||||
get_cursor_exec_config,
|
||||
get_optimizer_backend,
|
||||
get_target_backend,
|
||||
is_optimizer_chat_backend,
|
||||
@@ -53,6 +55,10 @@ def set_backend(name: str | None) -> str:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend(normalized)
|
||||
return normalized
|
||||
if normalized in {"cursor", "cursor_agent", "cursor_exec"}:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend("cursor_exec")
|
||||
return "cursor_exec"
|
||||
if normalized in {"qwen", "qwen_chat"}:
|
||||
set_optimizer_backend("openai_chat")
|
||||
set_target_backend("qwen_chat")
|
||||
@@ -84,6 +90,8 @@ def get_backend_name() -> str:
|
||||
return "qwen_chat"
|
||||
if optimizer == "openai_chat" and target == "minimax_chat":
|
||||
return "minimax_chat"
|
||||
if optimizer == "openai_chat" and target == "cursor_exec":
|
||||
return "cursor_exec"
|
||||
if optimizer == "openai_compatible" and target == "openai_compatible":
|
||||
return "openai_compatible"
|
||||
return f"{optimizer}+{target}"
|
||||
|
||||
@@ -28,6 +28,8 @@ CLAUDE_CODE_EXEC_PATH = os.environ.get("CLAUDE_CODE_EXEC_PATH", "claude")
|
||||
CLAUDE_CODE_EXEC_PROFILE = os.environ.get("CLAUDE_CODE_EXEC_PROFILE", "")
|
||||
CLAUDE_CODE_EXEC_USE_SDK = os.environ.get("CLAUDE_CODE_EXEC_USE_SDK", "auto")
|
||||
CLAUDE_CODE_EXEC_EFFORT = os.environ.get("CLAUDE_CODE_EXEC_EFFORT", "medium")
|
||||
CURSOR_EXEC_PATH = os.environ.get("CURSOR_EXEC_PATH", "cursor-agent")
|
||||
CURSOR_EXEC_SANDBOX = os.environ.get("CURSOR_EXEC_SANDBOX", "enabled")
|
||||
|
||||
|
||||
def _parse_int(value: str | None, default: int) -> int:
|
||||
@@ -72,10 +74,11 @@ def get_optimizer_backend() -> str:
|
||||
def set_target_backend(backend: str) -> None:
|
||||
global TARGET_BACKEND
|
||||
TARGET_BACKEND = normalize_backend_name(backend or "openai_chat")
|
||||
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "openai_compatible", "codex_exec", "claude_code_exec"}:
|
||||
if TARGET_BACKEND not in {"openai_chat", "claude_chat", "qwen_chat", "minimax_chat", "openai_compatible", "codex_exec", "claude_code_exec", "cursor_exec"}:
|
||||
raise ValueError(
|
||||
f"Unsupported target backend: {TARGET_BACKEND!r}. "
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', 'openai_compatible', 'codex_exec', and 'claude_code_exec'."
|
||||
"Supported values are 'openai_chat', 'claude_chat', 'qwen_chat', 'minimax_chat', "
|
||||
"'openai_compatible', 'codex_exec', 'claude_code_exec', and 'cursor_exec'."
|
||||
)
|
||||
os.environ["TARGET_BACKEND"] = TARGET_BACKEND
|
||||
|
||||
@@ -85,7 +88,7 @@ def get_target_backend() -> str:
|
||||
|
||||
|
||||
def is_target_exec_backend() -> bool:
|
||||
return TARGET_BACKEND in {"codex_exec", "claude_code_exec"}
|
||||
return TARGET_BACKEND in {"codex_exec", "claude_code_exec", "cursor_exec"}
|
||||
|
||||
|
||||
def is_optimizer_chat_backend() -> bool:
|
||||
@@ -198,3 +201,30 @@ def get_claude_code_exec_config() -> dict[str, str | int]:
|
||||
"max_thinking_tokens": CLAUDE_CODE_EXEC_MAX_THINKING_TOKENS,
|
||||
"empty_response_retries": EXEC_EMPTY_RESPONSE_RETRIES,
|
||||
}
|
||||
|
||||
|
||||
def configure_cursor_exec(
|
||||
*,
|
||||
path: str | None = None,
|
||||
sandbox: str | None = None,
|
||||
) -> None:
|
||||
global CURSOR_EXEC_PATH, CURSOR_EXEC_SANDBOX
|
||||
if path is not None:
|
||||
CURSOR_EXEC_PATH = str(path).strip() or "cursor-agent"
|
||||
os.environ["CURSOR_EXEC_PATH"] = CURSOR_EXEC_PATH
|
||||
if sandbox is not None:
|
||||
normalized_sandbox = str(sandbox).strip().lower() or "enabled"
|
||||
if normalized_sandbox not in {"enabled", "disabled"}:
|
||||
raise ValueError("cursor_exec sandbox must be 'enabled' or 'disabled'")
|
||||
CURSOR_EXEC_SANDBOX = normalized_sandbox
|
||||
os.environ["CURSOR_EXEC_SANDBOX"] = CURSOR_EXEC_SANDBOX
|
||||
|
||||
|
||||
def get_cursor_exec_config() -> dict[str, str | int]:
|
||||
if CURSOR_EXEC_SANDBOX not in {"enabled", "disabled"}:
|
||||
raise ValueError("cursor_exec sandbox must be 'enabled' or 'disabled'")
|
||||
return {
|
||||
"path": CURSOR_EXEC_PATH,
|
||||
"sandbox": CURSOR_EXEC_SANDBOX,
|
||||
"empty_response_retries": EXEC_EMPTY_RESPONSE_RETRIES,
|
||||
}
|
||||
|
||||
@@ -14,10 +14,10 @@ from typing import Any
|
||||
from skillopt.model.backend_config import (
|
||||
get_claude_code_exec_config,
|
||||
get_codex_exec_config,
|
||||
get_cursor_exec_config,
|
||||
get_target_backend,
|
||||
)
|
||||
|
||||
|
||||
ANSWER_SCHEMA: dict[str, Any] = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -227,6 +227,51 @@ def _build_claude_trace_summary(raw: str, response: str) -> str:
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _build_cursor_trace_summary(raw: str, response: str) -> str:
|
||||
model = ""
|
||||
permission_mode = ""
|
||||
session_id = ""
|
||||
duration_ms = 0
|
||||
tool_calls = 0
|
||||
terminal_error = False
|
||||
for line in (raw or "").splitlines():
|
||||
line = line.strip()
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if not isinstance(event, dict):
|
||||
continue
|
||||
if event.get("type") == "system" and event.get("subtype") == "init":
|
||||
model = str(event.get("model") or model)
|
||||
permission_mode = str(event.get("permissionMode") or permission_mode)
|
||||
session_id = str(event.get("session_id") or session_id)
|
||||
elif event.get("type") == "tool_call" and event.get("subtype") == "started":
|
||||
tool_calls += 1
|
||||
elif event.get("type") == "result":
|
||||
session_id = str(event.get("session_id") or session_id)
|
||||
try:
|
||||
duration_ms = int(event.get("duration_ms") or 0)
|
||||
except (TypeError, ValueError):
|
||||
duration_ms = 0
|
||||
terminal_error = bool(event.get("is_error")) or event.get("subtype") == "error"
|
||||
|
||||
parts = ["Cursor Agent Trace Summary"]
|
||||
if model:
|
||||
parts.append(f"- model: {model}")
|
||||
if permission_mode:
|
||||
parts.append(f"- permission mode: {permission_mode}")
|
||||
if session_id:
|
||||
parts.append(f"- session id: {session_id}")
|
||||
parts.append(f"- tool calls: {tool_calls}")
|
||||
parts.append(f"- duration ms: {duration_ms}")
|
||||
parts.append(f"- terminal error: {'yes' if terminal_error else 'no'}")
|
||||
parts.append(f"- final response chars: {len(response or '')}")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _persist_artifacts(
|
||||
*,
|
||||
work_dir: str,
|
||||
@@ -271,6 +316,16 @@ def _persist_claude_artifacts(work_dir: str, raw: str, response: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _persist_cursor_artifacts(work_dir: str, raw: str, response: str) -> None:
|
||||
_persist_artifacts(
|
||||
work_dir=work_dir,
|
||||
raw=_sanitize_cursor_trace(raw, preserve_markers=True),
|
||||
response=response,
|
||||
prefix="cursor",
|
||||
summary_builder=_build_cursor_trace_summary,
|
||||
)
|
||||
|
||||
|
||||
def parse_codex_raw(raw: str) -> dict:
|
||||
"""Parse raw Codex CLI output into step sections.
|
||||
|
||||
@@ -1016,6 +1071,239 @@ def run_codex_exec(
|
||||
return last_response, combined
|
||||
|
||||
|
||||
_CURSOR_SECRET_ASSIGNMENT = re.compile(
|
||||
r"(?i)\b(cursor_api_key|api[_ -]?key|authorization|bearer|"
|
||||
r"access[_ -]?token|refresh[_ -]?token|token|password)\b"
|
||||
r"(\s*[:=]\s*|\s+)(?:bearer\s+)?([^\s,;]+)"
|
||||
)
|
||||
_CURSOR_SECRET_TOKEN = re.compile(r"\b(?:sk|key)[_-][A-Za-z0-9_-]{8,}\b")
|
||||
_CURSOR_OMITTED_TRACE_FIELDS = {"args", "content", "filetext", "prompt", "result"}
|
||||
_CURSOR_SECRET_TRACE_FIELDS = {
|
||||
"accesstoken",
|
||||
"apikey",
|
||||
"authorization",
|
||||
"cursorapikey",
|
||||
"password",
|
||||
"refreshtoken",
|
||||
"secret",
|
||||
"token",
|
||||
}
|
||||
|
||||
|
||||
def _redact_cursor_error(value: str) -> str:
|
||||
text = _CURSOR_SECRET_ASSIGNMENT.sub(r"\1\2[REDACTED]", value or "")
|
||||
return _CURSOR_SECRET_TOKEN.sub("[REDACTED]", text)
|
||||
|
||||
|
||||
def _sanitize_cursor_json(value: Any, *, field: str = "") -> Any:
|
||||
normalized_field = re.sub(r"[^a-z0-9]", "", field.lower())
|
||||
if normalized_field in _CURSOR_OMITTED_TRACE_FIELDS:
|
||||
return "[OMITTED]"
|
||||
if (
|
||||
normalized_field in _CURSOR_SECRET_TRACE_FIELDS
|
||||
or normalized_field.endswith("apikey")
|
||||
):
|
||||
return "[REDACTED]"
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
str(key): _sanitize_cursor_json(item, field=str(key))
|
||||
for key, item in value.items()
|
||||
}
|
||||
if isinstance(value, list):
|
||||
return [_sanitize_cursor_json(item) for item in value]
|
||||
if isinstance(value, str):
|
||||
return _redact_cursor_error(value)
|
||||
return value
|
||||
|
||||
|
||||
def _cursor_process_text(value: str | bytes) -> str:
|
||||
if isinstance(value, bytes):
|
||||
return value.decode("utf-8", errors="replace")
|
||||
return str(value or "")
|
||||
|
||||
|
||||
def _sanitize_cursor_trace(
|
||||
raw: str | bytes,
|
||||
*,
|
||||
preserve_markers: bool = False,
|
||||
) -> str:
|
||||
text = _cursor_process_text(raw)
|
||||
sanitized: list[str] = []
|
||||
in_stderr = False
|
||||
omitted_stdout = False
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if preserve_markers and stripped.startswith("===== CURSOR CLI ATTEMPT "):
|
||||
in_stderr = False
|
||||
omitted_stdout = False
|
||||
sanitized.append(line)
|
||||
continue
|
||||
if preserve_markers and stripped == "[stderr]":
|
||||
in_stderr = True
|
||||
sanitized.append(line)
|
||||
continue
|
||||
if in_stderr:
|
||||
sanitized.append(_redact_cursor_error(line))
|
||||
continue
|
||||
if stripped.startswith("{"):
|
||||
try:
|
||||
event = json.loads(stripped)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
else:
|
||||
sanitized.append(
|
||||
json.dumps(
|
||||
_sanitize_cursor_json(event),
|
||||
ensure_ascii=False,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
)
|
||||
continue
|
||||
if not stripped:
|
||||
sanitized.append("")
|
||||
elif not omitted_stdout:
|
||||
sanitized.append("[OMITTED NON-JSON OUTPUT]")
|
||||
omitted_stdout = True
|
||||
return "\n".join(sanitized)
|
||||
|
||||
|
||||
def _parse_cursor_terminal(raw: str) -> tuple[str, str]:
|
||||
terminal: dict[str, Any] | None = None
|
||||
for line in (raw or "").splitlines():
|
||||
line = line.strip()
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
event = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(event, dict) and event.get("type") == "result":
|
||||
terminal = event
|
||||
|
||||
if terminal is None:
|
||||
return "", "Cursor Agent did not emit a terminal result"
|
||||
if terminal.get("is_error") is True or terminal.get("subtype") == "error":
|
||||
detail = str(terminal.get("result") or terminal.get("error") or "unknown error")
|
||||
return "", f"Cursor Agent returned an error result: {_redact_cursor_error(detail)}"
|
||||
result = terminal.get("result")
|
||||
if not isinstance(result, str) or not result.strip():
|
||||
return "", "Cursor Agent returned an empty terminal result"
|
||||
return result.strip(), ""
|
||||
|
||||
|
||||
def run_cursor_exec(
|
||||
*,
|
||||
work_dir: str,
|
||||
prompt: str,
|
||||
model: str,
|
||||
timeout: int,
|
||||
images: list[str] | None = None,
|
||||
data_dirs: list[str] | None = None,
|
||||
sandbox: str | None = None,
|
||||
allow_file_edits: bool = False,
|
||||
) -> tuple[str, str]:
|
||||
"""Run Cursor Agent headlessly as a benchmark target."""
|
||||
config = get_cursor_exec_config()
|
||||
retries = int(config.get("empty_response_retries", 0) or 0)
|
||||
add_dirs = _validated_add_dirs(work_dir, data_dirs, images)[1:]
|
||||
all_raw: list[str] = []
|
||||
last_error = "Cursor Agent returned no response"
|
||||
actual_sandbox = str(sandbox or config["sandbox"])
|
||||
if actual_sandbox not in {"enabled", "disabled"}:
|
||||
raise ValueError("Cursor Agent sandbox must be 'enabled' or 'disabled'")
|
||||
if allow_file_edits and actual_sandbox == "disabled":
|
||||
raise ValueError(
|
||||
"Cursor Agent file-edit rollouts require sandbox='enabled'; "
|
||||
"refusing to combine --force with a disabled sandbox"
|
||||
)
|
||||
|
||||
for attempt in range(retries + 1):
|
||||
attempt_prompt = _exec_prompt(
|
||||
_retry_prompt(prompt, attempt),
|
||||
allow_file_edits=allow_file_edits,
|
||||
)
|
||||
cmd = [
|
||||
str(config["path"]),
|
||||
"-p",
|
||||
"--output-format",
|
||||
"stream-json",
|
||||
"--trust",
|
||||
"--workspace",
|
||||
work_dir,
|
||||
"--sandbox",
|
||||
actual_sandbox,
|
||||
]
|
||||
if allow_file_edits:
|
||||
cmd.append("--force")
|
||||
else:
|
||||
cmd.extend(["--mode", "ask"])
|
||||
if model:
|
||||
cmd.extend(["--model", model])
|
||||
for path in add_dirs:
|
||||
cmd.extend(["--add-dir", path])
|
||||
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd,
|
||||
cwd=work_dir,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
input=attempt_prompt,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
stdout = exc.stdout or ""
|
||||
stderr = exc.stderr or ""
|
||||
raw = stdout
|
||||
safe_raw = _sanitize_cursor_trace(raw)
|
||||
if stderr:
|
||||
safe_stderr = _redact_cursor_error(_cursor_process_text(stderr))
|
||||
safe_raw = (
|
||||
f"{safe_raw}\n[stderr]\n{safe_stderr}"
|
||||
if safe_raw
|
||||
else f"[stderr]\n{safe_stderr}"
|
||||
)
|
||||
all_raw.append(f"===== CURSOR CLI ATTEMPT {attempt + 1} =====\n{safe_raw}")
|
||||
_persist_cursor_artifacts(work_dir, "\n\n".join(all_raw), "")
|
||||
raise
|
||||
except OSError as exc:
|
||||
detail = _redact_cursor_error(str(exc))
|
||||
raise RuntimeError(f"Cursor Agent could not be executed: {detail}") from exc
|
||||
|
||||
stdout = proc.stdout or ""
|
||||
stderr = proc.stderr or ""
|
||||
raw = stdout
|
||||
safe_raw = _sanitize_cursor_trace(raw)
|
||||
if stderr:
|
||||
safe_stderr = _redact_cursor_error(_cursor_process_text(stderr))
|
||||
safe_raw = (
|
||||
f"{safe_raw}\n[stderr]\n{safe_stderr}"
|
||||
if safe_raw
|
||||
else f"[stderr]\n{safe_stderr}"
|
||||
)
|
||||
all_raw.append(f"===== CURSOR CLI ATTEMPT {attempt + 1} =====\n{safe_raw}")
|
||||
combined = "\n\n".join(all_raw)
|
||||
|
||||
if proc.returncode != 0:
|
||||
_persist_cursor_artifacts(work_dir, combined, "")
|
||||
detail = _redact_cursor_error((stderr or stdout).strip())[:4000]
|
||||
raise RuntimeError(
|
||||
f"Cursor Agent failed with exit code {proc.returncode}: {detail}"
|
||||
)
|
||||
|
||||
response, last_error = _parse_cursor_terminal(stdout)
|
||||
if response:
|
||||
_persist_cursor_artifacts(work_dir, combined, response)
|
||||
return response, combined
|
||||
if last_error.startswith("Cursor Agent returned an error result"):
|
||||
_persist_cursor_artifacts(work_dir, combined, "")
|
||||
raise RuntimeError(last_error)
|
||||
|
||||
combined = "\n\n".join(all_raw)
|
||||
_persist_cursor_artifacts(work_dir, combined, "")
|
||||
raise RuntimeError(last_error)
|
||||
|
||||
|
||||
def run_target_exec(
|
||||
*,
|
||||
work_dir: str,
|
||||
@@ -1054,4 +1342,15 @@ def run_target_exec(
|
||||
permission_mode=permission_mode,
|
||||
allow_file_edits=allow_file_edits,
|
||||
)
|
||||
if backend == "cursor_exec":
|
||||
return run_cursor_exec(
|
||||
work_dir=work_dir,
|
||||
prompt=prompt,
|
||||
model=model,
|
||||
timeout=timeout,
|
||||
images=images,
|
||||
data_dirs=data_dirs,
|
||||
sandbox=sandbox,
|
||||
allow_file_edits=allow_file_edits,
|
||||
)
|
||||
raise ValueError(f"Unsupported exec backend: {backend}")
|
||||
|
||||
@@ -24,6 +24,7 @@ _BACKEND_DEFAULT_MODELS = {
|
||||
"claude": "claude-sonnet-4-6",
|
||||
"claude_chat": "claude-sonnet-4-6",
|
||||
"claude_code_exec": "claude-sonnet-4-6",
|
||||
"cursor_exec": "composer-2.5",
|
||||
"qwen_chat": "Qwen/Qwen3.5-4B",
|
||||
"minimax_chat": "MiniMax-M2.7",
|
||||
"openai_compatible": "gpt-4o-mini",
|
||||
@@ -40,6 +41,9 @@ _BACKEND_ALIASES = {
|
||||
"claude": "claude_chat",
|
||||
"claude_chat": "claude_chat",
|
||||
"claude_code_exec": "claude_code_exec",
|
||||
"cursor": "cursor_exec",
|
||||
"cursor_agent": "cursor_exec",
|
||||
"cursor_exec": "cursor_exec",
|
||||
"anthropic": "claude_chat",
|
||||
"qwen": "qwen_chat",
|
||||
"qwen_chat": "qwen_chat",
|
||||
|
||||
438
tests/test_cursor_exec_backend.py
Normal file
438
tests/test_cursor_exec_backend.py
Normal file
@@ -0,0 +1,438 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
import skillopt.model as model
|
||||
from skillopt.config import flatten_config
|
||||
from skillopt.model import backend_config
|
||||
from skillopt.model import codex_harness as harness
|
||||
from skillopt.model.common import default_model_for_backend, normalize_backend_name
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def restore_backend_state() -> Iterator[None]:
|
||||
optimizer_backend = backend_config.get_optimizer_backend()
|
||||
target_backend = backend_config.get_target_backend()
|
||||
cursor_path = backend_config.CURSOR_EXEC_PATH
|
||||
cursor_sandbox = backend_config.CURSOR_EXEC_SANDBOX
|
||||
retries = backend_config.EXEC_EMPTY_RESPONSE_RETRIES
|
||||
env = {
|
||||
key: os.environ.get(key)
|
||||
for key in (
|
||||
"OPTIMIZER_BACKEND",
|
||||
"TARGET_BACKEND",
|
||||
"CURSOR_EXEC_PATH",
|
||||
"CURSOR_EXEC_SANDBOX",
|
||||
)
|
||||
}
|
||||
yield
|
||||
backend_config.OPTIMIZER_BACKEND = optimizer_backend
|
||||
backend_config.TARGET_BACKEND = target_backend
|
||||
backend_config.CURSOR_EXEC_PATH = cursor_path
|
||||
backend_config.CURSOR_EXEC_SANDBOX = cursor_sandbox
|
||||
backend_config.EXEC_EMPTY_RESPONSE_RETRIES = retries
|
||||
for key, value in env.items():
|
||||
if value is None:
|
||||
os.environ.pop(key, None)
|
||||
else:
|
||||
os.environ[key] = value
|
||||
|
||||
|
||||
def _result(text: str = "<answer>A</answer>") -> str:
|
||||
return (
|
||||
'{"type":"system","subtype":"init","model":"composer-2.5",'
|
||||
'"permissionMode":"default","session_id":"session-1"}\n'
|
||||
'{"type":"tool_call","subtype":"started","call_id":"call-1",'
|
||||
'"tool_call":{"readToolCall":{"args":{"path":"task.md"}}}}\n'
|
||||
f'{{"type":"result","subtype":"success","is_error":false,'
|
||||
f'"duration_ms":12,"result":"{text}","session_id":"session-1"}}\n'
|
||||
)
|
||||
|
||||
|
||||
def _workspace(tmp_path: Path) -> Path:
|
||||
work_dir = tmp_path / "predictions" / "task-1" / "cursor_exec"
|
||||
work_dir.mkdir(parents=True)
|
||||
return work_dir
|
||||
|
||||
|
||||
def test_cursor_exec_is_target_only() -> None:
|
||||
backend_config.set_target_backend("cursor")
|
||||
|
||||
assert backend_config.get_target_backend() == "cursor_exec"
|
||||
assert backend_config.is_target_exec_backend()
|
||||
with pytest.raises(ValueError, match="Unsupported optimizer backend"):
|
||||
backend_config.set_optimizer_backend("cursor_exec")
|
||||
with pytest.raises(NotImplementedError, match="Exec backends"):
|
||||
model.chat_target("system", "user")
|
||||
|
||||
|
||||
def test_cursor_alias_and_default_model() -> None:
|
||||
assert normalize_backend_name("cursor") == "cursor_exec"
|
||||
assert normalize_backend_name("cursor_agent") == "cursor_exec"
|
||||
assert default_model_for_backend("cursor_exec") == "composer-2.5"
|
||||
|
||||
assert model.set_backend("cursor") == "cursor_exec"
|
||||
assert backend_config.get_optimizer_backend() == "openai_chat"
|
||||
assert backend_config.get_target_backend() == "cursor_exec"
|
||||
assert model.get_backend_name() == "cursor_exec"
|
||||
|
||||
|
||||
def test_cursor_config_flattens_and_validates() -> None:
|
||||
flat = flatten_config(
|
||||
{
|
||||
"model": {
|
||||
"cursor_exec_path": "/opt/cursor-agent",
|
||||
"cursor_exec_sandbox": "disabled",
|
||||
}
|
||||
}
|
||||
)
|
||||
assert flat["cursor_exec_path"] == "/opt/cursor-agent"
|
||||
assert flat["cursor_exec_sandbox"] == "disabled"
|
||||
|
||||
backend_config.configure_cursor_exec(path="cursor-test", sandbox="disabled")
|
||||
assert backend_config.get_cursor_exec_config()["path"] == "cursor-test"
|
||||
assert backend_config.get_cursor_exec_config()["sandbox"] == "disabled"
|
||||
with pytest.raises(ValueError, match="sandbox must be"):
|
||||
backend_config.configure_cursor_exec(sandbox="invalid")
|
||||
|
||||
|
||||
def test_train_cursor_shorthand_configures_target_only(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
from scripts import train
|
||||
|
||||
config_path = Path(__file__).parents[1] / "configs" / "_base_" / "default.yaml"
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
["train.py", "--config", str(config_path), "--backend", "cursor"],
|
||||
)
|
||||
|
||||
cfg = train.load_config(train.parse_args())
|
||||
|
||||
assert cfg["optimizer_backend"] == "openai_chat"
|
||||
assert cfg["target_backend"] == "cursor_exec"
|
||||
assert cfg["target_model"] == "composer-2.5"
|
||||
|
||||
|
||||
def test_read_only_cursor_exec_uses_stdin_and_preserves_trace(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
data_dir = tmp_path / "corpus"
|
||||
data_dir.mkdir()
|
||||
calls: list[tuple[list[str], dict[str, Any]]] = []
|
||||
|
||||
def fake_run(cmd: list[str], **kwargs: Any) -> SimpleNamespace:
|
||||
calls.append((cmd, kwargs))
|
||||
return SimpleNamespace(returncode=0, stdout="not json\n" + _result(), stderr="")
|
||||
|
||||
backend_config.configure_cursor_exec(path="cursor-test", sandbox="enabled")
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
response, raw = harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Answer the benchmark task.",
|
||||
model="composer-2.5",
|
||||
timeout=17,
|
||||
data_dirs=[str(data_dir)],
|
||||
)
|
||||
|
||||
assert response == "<answer>A</answer>"
|
||||
assert "tool_call" in raw
|
||||
assert "not json" not in raw
|
||||
assert "task.md" not in raw
|
||||
assert "<answer>A</answer>" not in raw
|
||||
cmd, kwargs = calls[0]
|
||||
assert cmd[:4] == ["cursor-test", "-p", "--output-format", "stream-json"]
|
||||
assert ["--mode", "ask"] == cmd[cmd.index("--mode"):cmd.index("--mode") + 2]
|
||||
assert "--force" not in cmd
|
||||
assert cmd[cmd.index("--workspace") + 1] == str(work_dir)
|
||||
assert cmd[cmd.index("--sandbox") + 1] == "enabled"
|
||||
assert cmd[cmd.index("--model") + 1] == "composer-2.5"
|
||||
assert cmd[cmd.index("--add-dir") + 1] == str(data_dir)
|
||||
assert kwargs["cwd"] == str(work_dir)
|
||||
assert kwargs["timeout"] == 17
|
||||
assert ".agents/skills/skillopt-target/SKILL.md" in kwargs["input"]
|
||||
assert "Do not modify files" in kwargs["input"]
|
||||
persisted_raw = (work_dir.parent / "cursor_raw.txt").read_text()
|
||||
assert "not json" not in persisted_raw
|
||||
assert "task.md" not in persisted_raw
|
||||
assert "<answer>A</answer>" not in persisted_raw
|
||||
summary = (work_dir.parent / "cursor_trace_summary.txt").read_text()
|
||||
assert "tool calls: 1" in summary
|
||||
assert "session-1" in summary
|
||||
|
||||
|
||||
def test_cursor_exec_force_is_limited_to_file_edit_rollouts(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
calls: list[tuple[list[str], str]] = []
|
||||
|
||||
def fake_run(cmd: list[str], **kwargs: Any) -> SimpleNamespace:
|
||||
calls.append((cmd, kwargs["input"]))
|
||||
return SimpleNamespace(returncode=0, stdout=_result(), stderr="")
|
||||
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Write solution.py.",
|
||||
model="",
|
||||
timeout=10,
|
||||
allow_file_edits=True,
|
||||
)
|
||||
|
||||
cmd, prompt = calls[0]
|
||||
assert "--force" in cmd
|
||||
assert "--mode" not in cmd
|
||||
assert "You may modify files" in prompt
|
||||
|
||||
|
||||
def test_cursor_exec_rejects_file_edits_with_disabled_sandbox(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
calls = 0
|
||||
|
||||
def fake_run(_cmd: list[str], **_kwargs: Any) -> SimpleNamespace:
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return SimpleNamespace(returncode=0, stdout=_result(), stderr="")
|
||||
|
||||
backend_config.configure_cursor_exec(sandbox="disabled")
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
with pytest.raises(ValueError, match="refusing to combine --force"):
|
||||
harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Write solution.py.",
|
||||
model="composer-2.5",
|
||||
timeout=10,
|
||||
allow_file_edits=True,
|
||||
)
|
||||
|
||||
assert calls == 0
|
||||
|
||||
|
||||
def test_cursor_exec_retries_zero_exit_malformed_output(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
prompts: list[str] = []
|
||||
|
||||
def fake_run(_cmd: list[str], **kwargs: Any) -> SimpleNamespace:
|
||||
prompts.append(kwargs["input"])
|
||||
stdout = "malformed output" if len(prompts) == 1 else _result()
|
||||
return SimpleNamespace(returncode=0, stdout=stdout, stderr="")
|
||||
|
||||
backend_config.EXEC_EMPTY_RESPONSE_RETRIES = 1
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
response, _raw = harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Answer.",
|
||||
model="",
|
||||
timeout=10,
|
||||
)
|
||||
|
||||
assert response == "<answer>A</answer>"
|
||||
assert len(prompts) == 2
|
||||
assert "Previous execution returned an empty final response" in prompts[1]
|
||||
|
||||
|
||||
def test_cursor_trace_summary_tolerates_malformed_metadata() -> None:
|
||||
raw = (
|
||||
'{"type":"result","subtype":"success","is_error":false,'
|
||||
'"duration_ms":"unknown","result":"done"}\n'
|
||||
)
|
||||
|
||||
summary = harness._build_cursor_trace_summary(raw, "done")
|
||||
|
||||
assert "duration ms: 0" in summary
|
||||
|
||||
|
||||
def test_cursor_trace_omits_message_and_tool_payloads() -> None:
|
||||
raw = "\n".join(
|
||||
[
|
||||
'{"type":"user","message":{"role":"user","content":'
|
||||
'[{"type":"text","text":"private prompt"}]}}',
|
||||
'{"type":"assistant","message":{"role":"assistant","content":'
|
||||
'[{"type":"text","text":"private response"}]}}',
|
||||
'{"type":"tool_call","subtype":"completed","tool_call":'
|
||||
'{"readToolCall":{"args":{"path":"secret.txt"},"result":'
|
||||
'{"success":{"content":"private file contents"}}}}}',
|
||||
'{"type":"result","subtype":"success","is_error":false,'
|
||||
'"duration_ms":1,"result":"private final answer"}',
|
||||
]
|
||||
)
|
||||
|
||||
sanitized = harness._sanitize_cursor_trace(raw)
|
||||
events = [json.loads(line) for line in sanitized.splitlines()]
|
||||
|
||||
assert events[0]["message"]["content"] == "[OMITTED]"
|
||||
assert events[1]["message"]["content"] == "[OMITTED]"
|
||||
assert events[2]["tool_call"]["readToolCall"]["args"] == "[OMITTED]"
|
||||
assert events[2]["tool_call"]["readToolCall"]["result"] == "[OMITTED]"
|
||||
assert events[3]["result"] == "[OMITTED]"
|
||||
assert "private" not in sanitized
|
||||
assert "secret.txt" not in sanitized
|
||||
|
||||
|
||||
def test_cursor_exec_does_not_retry_error_result_and_redacts_detail(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
calls = 0
|
||||
|
||||
def fake_run(_cmd: list[str], **_kwargs: Any) -> SimpleNamespace:
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
stdout = (
|
||||
'{"type":"result","subtype":"error","is_error":true,'
|
||||
'"result":"API key: cursor-secret-value"}\n'
|
||||
)
|
||||
return SimpleNamespace(returncode=0, stdout=stdout, stderr="")
|
||||
|
||||
backend_config.EXEC_EMPTY_RESPONSE_RETRIES = 1
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Answer.",
|
||||
model="",
|
||||
timeout=10,
|
||||
)
|
||||
|
||||
assert calls == 1
|
||||
assert "[REDACTED]" in str(exc_info.value)
|
||||
assert "cursor-secret-value" not in str(exc_info.value)
|
||||
persisted_raw = (work_dir.parent / "cursor_raw.txt").read_text()
|
||||
assert "cursor-secret-value" not in persisted_raw
|
||||
assert '"result":"[OMITTED]"' in persisted_raw
|
||||
|
||||
|
||||
def test_cursor_exec_nonzero_exit_is_not_retried(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
calls = 0
|
||||
|
||||
def fake_run(_cmd: list[str], **_kwargs: Any) -> SimpleNamespace:
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
return SimpleNamespace(
|
||||
returncode=1,
|
||||
stdout="",
|
||||
stderr="CURSOR_API_KEY=cursor-secret-token authentication failed",
|
||||
)
|
||||
|
||||
backend_config.EXEC_EMPTY_RESPONSE_RETRIES = 1
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
with pytest.raises(RuntimeError) as exc_info:
|
||||
harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Answer.",
|
||||
model="",
|
||||
timeout=10,
|
||||
)
|
||||
|
||||
assert calls == 1
|
||||
assert "[REDACTED]" in str(exc_info.value)
|
||||
assert "cursor-secret-token" not in str(exc_info.value)
|
||||
persisted_raw = (work_dir.parent / "cursor_raw.txt").read_text()
|
||||
assert "cursor-secret-token" not in persisted_raw
|
||||
assert "CURSOR_API_KEY=[REDACTED]" in persisted_raw
|
||||
|
||||
|
||||
def test_cursor_exec_timeout_is_persisted(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
|
||||
def fake_run(_cmd: list[str], **_kwargs: Any) -> SimpleNamespace:
|
||||
raise subprocess.TimeoutExpired(
|
||||
"cursor-test",
|
||||
3,
|
||||
output=b"partial CURSOR_API_KEY=timeout-secret",
|
||||
)
|
||||
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
with pytest.raises(subprocess.TimeoutExpired):
|
||||
harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Answer.",
|
||||
model="",
|
||||
timeout=3,
|
||||
)
|
||||
|
||||
persisted_raw = (work_dir.parent / "cursor_raw.txt").read_text()
|
||||
assert "partial" not in persisted_raw
|
||||
assert "[OMITTED NON-JSON OUTPUT]" in persisted_raw
|
||||
assert "timeout-secret" not in persisted_raw
|
||||
|
||||
|
||||
def test_cursor_exec_spawn_failure_is_actionable(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
work_dir = _workspace(tmp_path)
|
||||
|
||||
def fake_run(_cmd: list[str], **_kwargs: Any) -> SimpleNamespace:
|
||||
raise FileNotFoundError("cursor-agent-test was not found")
|
||||
|
||||
monkeypatch.setattr(harness.subprocess, "run", fake_run)
|
||||
|
||||
with pytest.raises(RuntimeError, match="could not be executed"):
|
||||
harness.run_cursor_exec(
|
||||
work_dir=str(work_dir),
|
||||
prompt="Answer.",
|
||||
model="",
|
||||
timeout=3,
|
||||
)
|
||||
|
||||
|
||||
def test_run_target_exec_dispatches_cursor(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
def fake_cursor(**kwargs: Any) -> tuple[str, str]:
|
||||
captured.update(kwargs)
|
||||
return "cursor response", "cursor trace"
|
||||
|
||||
backend_config.set_target_backend("cursor_exec")
|
||||
monkeypatch.setattr(harness, "run_cursor_exec", fake_cursor)
|
||||
|
||||
response, raw = harness.run_target_exec(
|
||||
work_dir=str(tmp_path),
|
||||
prompt="task",
|
||||
model="composer-2.5",
|
||||
timeout=20,
|
||||
allow_file_edits=True,
|
||||
)
|
||||
|
||||
assert (response, raw) == ("cursor response", "cursor trace")
|
||||
assert captured["allow_file_edits"] is True
|
||||
assert captured["model"] == "composer-2.5"
|
||||
Reference in New Issue
Block a user