mirror of https://github.com/microsoft/SkillOpt.git synced 2026-07-07 00:15:39 +08:00

Files

Shunsuke 98d0430bee refactor: make EnvAdapter.reflect a shared default (fixes dropped reflect kwargs)

All six adapters duplicated an identical reflect() that delegates to
run_minibatch_reflect. The copies had drifted: OfficeQA/DocVQA silently
dropped meta_skill_context and ALFWorld dropped update_mode, so those
analysts ran without inputs every other benchmark receives (active under
the default use_meta_skill: true).

Move the delegation into EnvAdapter.reflect as one default that forwards
all kwargs uniformly, and delete the six overrides. reflect is no longer
abstract — adapters inherit it and override only for custom logic.

Net -225 lines. Behavior change: OfficeQA/DocVQA/ALFWorld reflect now
receive the kwargs they previously dropped; the three already-correct
benchmarks are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-15 09:06:00 +00:00

config_template.yaml

envs/_template: make template instantiable against real EnvAdapter ABC

2026-06-01 20:15:12 +00:00

env_template.py

refactor: make EnvAdapter.reflect a shared default (fixes dropped reflect kwargs)

2026-06-15 09:06:00 +00:00

loader_template.py

envs/_template: make template instantiable against real EnvAdapter ABC

2026-06-01 20:15:12 +00:00

README.md

refactor: make EnvAdapter.reflect a shared default (fixes dropped reflect kwargs)

2026-06-15 09:06:00 +00:00

README.md

Benchmark Template

This directory provides scaffold files for adding a new benchmark to SkillOpt.

Files

env_template.py — Environment adapter template (subclasses EnvAdapter; implements the 4 abstract methods so the file is instantiable out of the box — reflect is inherited).
loader_template.py — Data loader template (subclasses SplitDataLoader; implements load_split_items for .json/.jsonl).
config_template.yaml — Config file template.

Usage

Copy the directory:

cp -r skillopt/envs/_template skillopt/envs/your_benchmark

Rename the files (drop the _template suffix):
```
cd skillopt/envs/your_benchmark
mv env_template.py    adapter.py
mv loader_template.py dataloader.py
```
…and inside each file rename the classes (TemplateBenchmarkEnv → YourBenchmarkAdapter, TemplateBenchmarkLoader → YourBenchmarkLoader) and fix the cross-import in adapter.py.
Implement the TODO blocks inside adapter.py:rollout and the _normalize_item helper in dataloader.py. (reflect is inherited from EnvAdapter; override it only for custom reflection logic.)
Register the adapter — add a try / except ImportError block in scripts/train.py's _register_builtins() mapping the registry key to your YourBenchmarkAdapter class. There is no BENCHMARK_REGISTRY dict in skillopt/envs/__init__.py; the live registry is _ENV_REGISTRY in scripts/train.py.
Create the config at configs/your_benchmark/default.yaml (start from config_template.yaml). _base_ is a string path, not a list.

See the Add a New Benchmark guide for the full step-by-step with a worked docfaithful example.