diff --git a/README.md b/README.md index f81c37b..e7783e0 100644 --- a/README.md +++ b/README.md @@ -208,6 +208,23 @@ Re-running the same command auto-resumes from the last completed step. --- +## Community-contributed configs + +These are **not** default SkillOpt settings — they are reference configs +contributed by users for specific scenarios. The paper-reported numbers +were obtained with the default settings, not these. + +- **`configs/examples/soft_gate.yaml`** *(PR #25, contributed by + [@lvbaocheng](https://github.com/lvbaocheng))* — switches the + validation gate from exact-match (`hard`) to soft / partial-credit + (`soft` or `mixed`). Useful when the held-out **selection split is + small** (e.g. ≤ ~10 items) and the **reward is continuous**, where the + discrete hard gate often rejects every candidate and training stalls. + See the comment at the top of the file for details and when not to use + it. + +--- + ## WebUI Launch the monitoring dashboard (optional): diff --git a/configs/_base_/default.yaml b/configs/_base_/default.yaml index 82d9e0b..ed8f61d 100644 --- a/configs/_base_/default.yaml +++ b/configs/_base_/default.yaml @@ -71,12 +71,6 @@ optimizer: evaluation: use_gate: true - # gate_metric: 'hard' (default, backward-compatible), - # 'soft' (use soft/F1 score), - # 'mixed' ((1 - w) * hard + w * soft). - # See skillopt/evaluation/gate.py for details. - gate_metric: hard - gate_mixed_weight: 0.5 sel_env_num: 0 test_env_num: 0 eval_test: true diff --git a/configs/examples/soft_gate.yaml b/configs/examples/soft_gate.yaml new file mode 100644 index 0000000..2f83b3f --- /dev/null +++ b/configs/examples/soft_gate.yaml @@ -0,0 +1,47 @@ +# ───────────────────────────────────────────────────────────────────────────── +# Example: soft / mixed validation-gate metric (community-contributed, PR #25) +# ───────────────────────────────────────────────────────────────────────────── +# +# This is NOT a default SkillOpt setting and was NOT used to produce the +# numbers reported in the paper. It is provided as a reference for users +# who encounter a specific scenario where the default `hard` gate is too +# coarse to drive training. +# +# When to consider this: +# - You are running on a custom environment. +# - Your held-out *selection* split has very few items (e.g. ≤ ~10). +# - Your reward function is continuous / partial-credit (e.g. F1, BLEU, +# soft match) rather than purely binary 0/1. +# +# Symptom this addresses: +# With a small selection split + continuous rewards, candidate skills +# often improve per-item soft scores (e.g. 0.06 → 0.26 on one item) but +# never flip the discrete hard outcome. The default `hard` gate then +# rejects every candidate and training stalls. Switching the gate to +# `soft` or `mixed` lets these partial improvements be accepted. +# +# When NOT to use this: +# - When reproducing the paper. The paper-reported numbers were obtained +# under the default `hard` gate. +# - When your selection split is large (dozens+ items) and / or your +# reward is already binary — `hard` is the more conservative choice +# and matches the design described in the paper. +# +# To use: inherit your env config from this file, e.g. +# _base_: ../examples/soft_gate.yaml +# or copy the `evaluation:` block below into your config. +# ───────────────────────────────────────────────────────────────────────────── + +_base_: ../_base_/default.yaml + +evaluation: + # Three options: + # 'hard' — default; exact-match accuracy. Use this to reproduce the paper. + # 'soft' — per-item soft / partial-credit score (recommended for the + # small-split + continuous-reward scenario described above). + # 'mixed' — weighted average: (1 - w) * hard + w * soft, with `w` set by + # `gate_mixed_weight` below. + gate_metric: soft + + # Only used when gate_metric == 'mixed'. Ignored otherwise. + gate_mixed_weight: 0.5