microsoft-SkillOpt/configs/ablation_study/validation.md

Ablation Validation Checklist

Use this checklist before launch, during monitoring, and before filling docs/ablation_paper_tables.md.

Before Launch

Run from repo root:

cd /home/azureuser/workspace-gzy/SkillReflection
export ALFWORLD_DATA=/home/azureuser/.cache/alfworld

Verify syntax for edited files:

/home/azureuser/workspace-gzy/miniconda3/envs/reflact/bin/python -m py_compile \
  scripts/run_ablation_matrix.py \
  scripts/train.py \
  reflact/model/azure_openai.py \
  reflact/envs/searchqa/rollout.py \
  reflact/envs/spreadsheetbench/rollout.py \
  reflact/envs/livemathematicianbench/rollout.py \
  reflact/envs/alfworld/rollout.py \
  reflact/envs/docvqa/rollout.py

Check active runs and duplicate env.out_root before starting more:

/home/azureuser/workspace-gzy/miniconda3/envs/reflact/bin/python - <<'PY'
import subprocess, re, collections
try:
    raw = subprocess.check_output(["pgrep", "-af", "scripts/train.py"], text=True)
except subprocess.CalledProcessError:
    raw = ""
roots = []
for line in raw.splitlines():
    m = re.search(r"env\.out_root=([^\s]+)", line)
    if m:
        roots.append(m.group(1))
ctr = collections.Counter(roots)
print("train_count", len(roots))
print("duplicate_roots", [r.rsplit("/", 1)[-1] for r, c in ctr.items() if c > 1])
for root in sorted(roots):
    print(root.rsplit("/", 1)[-1])
PY

During Monitoring

Check launchers:

pgrep -af 'scripts/run_ablation_matrix.py' || true
tail -80 outputs/ablation_docvqa_20260503_160225_run/launcher_parallel8.log 2>/dev/null || true
tail -80 outputs/ablation_livemath_alfworld_clean_20260503_155155_run/launcher_livemath_parallel8.log 2>/dev/null || true
tail -80 outputs/ablation_livemath_alfworld_clean_20260503_155155_run/launcher_alfworld_parallel1.log 2>/dev/null || true

Scan current logs for new hard failures:

rg -n "Traceback|ERROR|Error code|AuthenticationError|BadRequest|RateLimit|content_filter|Killed|OutOfMemory|\\[FAIL\\]|\\[RETRY\\]" \
  outputs/ablation_docvqa_20260503_160225_run/logs \
  outputs/ablation_livemath_alfworld_clean_20260503_155155_run/logs \
  outputs/ablation_batch_searchqa_spreadsheet_20260503_153902_run/logs \
  -g '*.log' | tail -160 || true

Check resource pressure:

df -h /tmp
du -sh /tmp/ray 2>/dev/null || true
free -h | sed -n '1,3p'

Quality Checks

LiveMathBench current valid runs should not look like old 768/512 runs:

/home/azureuser/workspace-gzy/miniconda3/envs/reflact/bin/python - <<'PY'
import json, pathlib
root = pathlib.Path("outputs/ablation_livemath_alfworld_clean_20260503_155155_run")
for run in sorted(root.glob("*livemathematicianbench*")):
    if not run.is_dir() or "archive" in str(run):
        continue
    for rel in ["test_eval_baseline/results.jsonl", "test_eval/results.jsonl"]:
        p = run / rel
        if not p.exists():
            continue
        rows = [json.loads(l) for l in p.open(errors="ignore") if l.strip()]
        empty = sum(1 for r in rows if not str(r.get("response", "")).strip())
        answer = sum(1 for r in rows if "<answer>" in str(r.get("response", "")).lower())
        if empty:
            print(run.name, rel, "empty", empty, "answer", answer, "n", len(rows))
PY

ALFWorld valid runs must not contain empty action or missing action:

/home/azureuser/workspace-gzy/miniconda3/envs/reflact/bin/python - <<'PY'
import json, pathlib
root = pathlib.Path("outputs/ablation_livemath_alfworld_clean_20260503_155155_run")
for run in sorted(root.glob("*alfworld*")):
    if not run.is_dir() or "archive" in str(run):
        continue
    bad = []
    fallback = 0
    for c in run.glob("**/conversation.json"):
        data = json.load(c.open(errors="ignore"))
        for step in data:
            if step.get("step") is None:
                continue
            if not step.get("action"):
                bad.append(str(c.relative_to(run)))
                break
            mr = str(step.get("model_response", ""))
            if "empty model response" in mr or "missing action tag" in mr:
                fallback += 1
    print(run.name, "bad_action_files", len(bad), "fallback", fallback)
PY

Filling Tables

Use only summary.json fields:

• best_selection_hard -> Best Sel
• baseline_test_hard -> Base Test
• test_hard -> Best Test
• test_delta_hard -> Delta
• total_accepts -> Accept
• total_rejects -> Reject
• token_summary._total.total_tokens -> Tokens

Do not fill table rows from logs alone.