244e346b83
- Skill optimization framework with training loop analogy - 11 benchmarks, 4 model backends (Azure OpenAI, Claude, Codex, Qwen) - WebUI for browser-based training control - Pluggable architecture for extending benchmarks and backends
4.5 KiB
hide
| hide | |
|---|---|
|
SkillOpt
Train Agent Skills Like Neural Networks
Optimize natural-language skill documents through iterative rollout, reflection, and gated validation — with epochs, learning rates, and validation gates — without touching model weights.
Get Started :material-rocket-launch:{ .md-button .md-button--primary } View on GitHub :material-github:{ .md-button }
How It Works
Deep Learning Analogy
SkillOpt brings the familiar deep-learning training paradigm to agentic prompt optimization:
| Deep Learning | SkillOpt |
|---|---|
| Model weights | Skill document (Markdown) |
| Forward pass | Rollout (student executes tasks) |
| Loss / gradient | Reflect (teacher produces edit patches) |
| Gradient clipping | Edit selection (learning_rate = max edits) |
| SGD step | Patch application to skill |
| Validation set | Gated evaluation on selection split |
| LR schedule | lr_scheduler: cosine, linear, constant |
| Epochs | Multi-epoch with slow update & meta skill memory |
Supported Benchmarks
| Benchmark | Type | Config |
|---|---|---|
| DocVQA | Document QA | configs/docvqa/ |
| ALFWorld | Embodied AI | configs/alfworld/ |
| OfficeQA | Enterprise QA | configs/officeqa/ |
| SearchQA | Open-domain QA | configs/searchqa/ |
| LiveMathBench | Math reasoning | configs/livemathematicianbench/ |
| SWEBench | Software Engineering | configs/swebench/ |
| + 5 more | Various | See docs |
Quick Example
# Install
pip install -e .
# Configure credentials
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-key"
# Train on SearchQA
python scripts/train.py --config configs/searchqa/default.yaml
# Evaluate best skill
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/best_skill.md
-
:material-book-open-variant:{ .lg .middle } Getting Started
Install SkillOpt, configure your API keys, and run your first experiment in 5 minutes.
-
:material-puzzle:{ .lg .middle } Add a Benchmark
Extend SkillOpt with your own benchmark in ~100 lines of code.
-
:material-cog:{ .lg .middle } Configuration
Full reference for all hyperparameters with deep learning analogies.
-
:material-monitor-dashboard:{ .lg .middle } WebUI
Configure, launch, and monitor training from your browser.