mirror of https://github.com/microsoft/SkillOpt.git synced 2026-07-03 14:02:58 +08:00

Files

Yifan Yang 4eb4c64b2a envs/_template: make template instantiable against real EnvAdapter ABC

The shipped env_template.py and loader_template.py described the same
fictional async execute / evaluate / build_prompt API documented in
docs/reference/api.md. As a result TemplateBenchmarkEnv(cfg) raised
'TypeError: Can't instantiate abstract class' for every copy-and-paste
user who followed the in-tree scaffold.

Rewrite the template so it's a working starting point:

- env_template.py: TemplateBenchmarkEnv(EnvAdapter) now implements all
  five real abstract methods (build_train_env, build_eval_env, rollout,
  reflect, get_task_types) with no-op defaults documented as TODO.
  Instantiable today; pytest 60/60 still passes.
- loader_template.py: TemplateBenchmarkLoader(SplitDataLoader)
  implements load_split_items for .json / .jsonl input and explains the
  optional load_raw_items override for split_mode="ratio".
- README.md: usage steps now point at scripts/train.py's _ENV_REGISTRY
  (the real registry) instead of a non-existent BENCHMARK_REGISTRY in
  skillopt/envs/__init__.py, and link to the rewritten new-benchmark
  guide.
- config_template.yaml: _base_ is a string path (not a list, which the
  loader rejects); skill_init is commented out with a note so the
  template config doesn't reference a file the user hasn't created.

Verified locally: 'from skillopt.envs._template.env_template import
TemplateBenchmarkEnv; TemplateBenchmarkEnv()' succeeds. Refs
microsoft/SkillOpt#30.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>

2026-06-01 20:15:12 +00:00

config_template.yaml

envs/_template: make template instantiable against real EnvAdapter ABC

2026-06-01 20:15:12 +00:00

env_template.py

envs/_template: make template instantiable against real EnvAdapter ABC

2026-06-01 20:15:12 +00:00

loader_template.py

envs/_template: make template instantiable against real EnvAdapter ABC

2026-06-01 20:15:12 +00:00

README.md

envs/_template: make template instantiable against real EnvAdapter ABC

2026-06-01 20:15:12 +00:00

README.md

Benchmark Template

This directory provides scaffold files for adding a new benchmark to SkillOpt.

Files

env_template.py — Environment adapter template (subclasses EnvAdapter; implements the 5 abstract methods so the file is instantiable out of the box).
loader_template.py — Data loader template (subclasses SplitDataLoader; implements load_split_items for .json/.jsonl).
config_template.yaml — Config file template.

Usage

Copy the directory:

cp -r skillopt/envs/_template skillopt/envs/your_benchmark

Rename the files (drop the _template suffix):
```
cd skillopt/envs/your_benchmark
mv env_template.py    adapter.py
mv loader_template.py loader.py
```
…and inside each file rename the classes (TemplateBenchmarkEnv → YourBenchmarkAdapter, TemplateBenchmarkLoader → YourBenchmarkLoader) and fix the cross-import in adapter.py.
Implement the TODO blocks inside adapter.py:rollout and the _normalize_item helper in loader.py. If you want real reflection, uncomment the run_minibatch_reflect block in adapter.py:reflect.
Register the adapter — add a try / except ImportError block in scripts/train.py's _register_builtins() mapping the registry key to your YourBenchmarkAdapter class. There is no BENCHMARK_REGISTRY dict in skillopt/envs/__init__.py; the live registry is _ENV_REGISTRY in scripts/train.py.
Create the config at configs/your_benchmark/default.yaml (start from config_template.yaml). _base_ is a string path, not a list.

See the Add a New Benchmark guide for the full step-by-step with a worked docfaithful example.