From c1ac570d944ee7f83fc7c4273abfcb4bfdfea392 Mon Sep 17 00:00:00 2001 From: Cuzyoung Date: Wed, 10 Jun 2026 13:48:43 +0000 Subject: [PATCH] =?UTF-8?q?docs(guideline):=20make=20SearchQA=20the=20firs?= =?UTF-8?q?t=20demo=20=E2=80=94=20copy-paste=20materialization=20snippet?= =?UTF-8?q?=20+=20train=20command?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- docs/guideline.html | 58 +++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 20 deletions(-) diff --git a/docs/guideline.html b/docs/guideline.html index 4029e6d..ddc6567 100644 --- a/docs/guideline.html +++ b/docs/guideline.html @@ -445,30 +445,48 @@ skillopt/ # the package

What ships in this repo: ready-to-use configs and pretrained skills (ckpt/) for six benchmarks, plus lightweight ID manifests under data/. The manifests - list which examples each split uses but do not contain - the example contents — so for most benchmarks you materialize the data - once before training (see below).

-

Fastest out-of-the-box run — ALFWorld. Its bundled - split (data/alfworld_path_split) is directly usable; you - only need the ALFWorld game files:

-
pip install -e ".[alfworld]"
-alfworld-download
-export ALFWORLD_DATA=~/.cache/alfworld   # data root containing json_2.1.1
+      pin exactly which examples each split uses but do not
+      contain the example contents — so you materialize the data once before
+      the first run.

+      
Step 1 — materialize the SearchQA splits (one-time; downloads the ~6.5 GB source dataset). The manifest IDs match the key field of the
+      lucadiliello/searchqa
+      dataset:

+
pip install datasets
+python - <<'PY'
+import json, os
+from datasets import load_dataset
 
-python scripts/train.py \
-    --config configs/alfworld/default.yaml \
-    --split_dir data/alfworld_path_split \
+ds = load_dataset("lucadiliello/searchqa")
+by_key = {r["key"]: r for split in ds.values() for r in split}
+
+for split in ["train", "val", "test"]:
+    ids = json.load(open(f"data/searchqa_id_split/{split}/items.json"))
+    items = []
+    for x in ids:
+        r = by_key[x["id"]]
+        items.append({"id": r["key"], "question": r["question"],
+                      "context": r["context"], "answers": r["answers"]})
+    os.makedirs(f"data/searchqa_split/{split}", exist_ok=True)
+    json.dump(items, open(f"data/searchqa_split/{split}/items.json", "w"))
+    print(split, len(items))
+PY

+      
Step 2 — train (4 epochs × batch 40; see §3.2
+      for the CLI reference):

+
python scripts/train.py \
+    --config configs/searchqa/default.yaml \
+    --split_dir data/searchqa_split \
     --azure_openai_endpoint https://your-resource.openai.azure.com/ \
     --optimizer_model gpt-5.5 \
     --target_model gpt-5.5

-      
Other benchmarks (e.g. SearchQA) require a one-time
-      data materialization step: download the raw dataset from the source
-      listed in data/README.md,
-      match the manifest IDs to raw examples (the README documents the lookup
-      key per benchmark), and write the resulting
-      train/val/test item files into a split directory. Then run
-      the commands in §3.2 with --split_dir pointing at it. The
-      required item fields are documented in §4.2.

+      
Other benchmarks follow the same pattern — materialize from the raw
+      source listed in
+      data/README.md
+      (it documents the lookup key per benchmark), then point
+      --split_dir at the result. The one exception is
+      ALFWorld, whose bundled
+      data/alfworld_path_split works directly: just
+      pip install -e ".[alfworld]" && alfworld-download and
+      set $ALFWORLD_DATA.

       
To sanity-check your setup without training, evaluate a
       packaged pretrained skill instead (§3.3 uses
       ckpt/searchqa/gpt5.5_skill.md), or launch the monitoring