It is intentionally separate from the benchmark default configs. The benchmark configs under configs/<benchmark>/default.yaml remain the source task configs; this folder records the exact matrix-level overrides, run roots, launch commands, and validation rules used for the paper ablations.

Files

matrix.yaml: canonical ablation matrix, common overrides, benchmark splits, token/output caps, and invalid-run rules.
launch_commands.sh: exact launcher commands for the valid run roots.
validation.md: monitoring, result extraction, and invalidation checklist.

Source Of Truth

Use the matrix launcher:

/home/azureuser/workspace-gzy/miniconda3/envs/reflact/bin/python scripts/run_ablation_matrix.py

The launcher builds runs from the same defaults and values recorded in matrix.yaml. It skips completed runs by checking summary.json and skips active runs by checking env.out_root in active scripts/train.py processes.

Do not manually rerun a completed run into the same env.out_root. If a run is invalid, archive or remove its output directory first, then let the launcher start it cleanly.

Current Correct Run Roots

SearchQA / SpreadsheetBench original ablations: outputs/ablation_20260502_040604_unique48
SearchQA / SpreadsheetBench batch-size ablations: outputs/ablation_batch_searchqa_spreadsheet_20260503_153902_run
LiveMathBench / ALFWorld clean ablations: outputs/ablation_livemath_alfworld_clean_20260503_155155_run
DocVQA ablations: outputs/ablation_docvqa_20260503_160225_run

Archived, superseded, misaligned, dry-run, or pre-fix directories must not be used for paper tables.

End-To-End Runbook

Environment

Run from the repository root:

cd /home/azureuser/workspace-gzy/SkillReflection

Always use:

PY=/home/azureuser/workspace-gzy/miniconda3/envs/reflact/bin/python
export ALFWORLD_DATA=/home/azureuser/.cache/alfworld

Default model/auth settings are generated by scripts/run_ablation_matrix.py:

teacher=gpt-5.5
student=gpt-5.5
teacher_backend=openai_chat
student_backend=openai_chat
reasoning_effort=medium
teacher/student endpoint=https://t2vgoaigpt4o3.openai.azure.com/
teacher/student api_version=2024-12-01-preview
teacher/student auth_mode=azure_cli

Core training settings:

train.num_epochs=4
train.train_size=0
train.batch_size=40
train.accumulation=1
train.seed=42
gradient.minibatch_size=8
gradient.merge_batch_size=8
gradient.analyst_workers=16
gradient.use_deep_reflect=false
optimizer.learning_rate=4
optimizer.min_learning_rate=2
optimizer.lr_scheduler=cosine
optimizer.lr_control_mode=fixed
optimizer.use_slow_update=true
optimizer.slow_update_samples=20
optimizer.use_meta_skill=true
optimizer.use_meta_reflect=false
optimizer.longitudinal_pair_policy=mixed
evaluation.use_gate=true
evaluation.eval_test=true
env.split_mode=split_dir

train.train_size=0 is intentional. The dataloader derives the train size from the fixed split. Batch-size ablations rely on the default ceil(train_size / batch_size) behavior; the last batch can be smaller than train.batch_size.

Fixed Splits

Default split directories:

searchqa: data/ablation_splits/searchqa/2-1-7_seed42
spreadsheetbench: data/ablation_splits/spreadsheetbench/2-1-7_seed42
livemathematicianbench: data/ablation_splits/livemathematicianbench/2-1-7_seed42
alfworld: data/ablation_splits/alfworld/2-1-7_seed42
docvqa: /home/azureuser/zisu/SkillReflection/data/docvqa/splits

Default train/val/test sizes:

Benchmark	Train	Val	Test
SearchQA	400	200	1400
SpreadsheetBench	80	40	280
LiveMathBench	35	18	124
ALFWorld	39	18	134
DocVQA	1070	535	3744

DocVQA images are not copied. The valid setup uses:

data/docvqa_images -> /home/azureuser/zisu/SkillReflection/data/docvqa_images

2026-05-05 DocVQA data correction: all DocVQA final reruns should use the zisu 10% split above and a fresh output root such as outputs/ablation_docvqa_zisu10pct_20260505_run. The older local data/ablation_splits/docvqa/2-1-7_seed42 contains the same 5349 questionId pool but a different train/val/test assignment, so its completed summaries are historical only.

Matrix Groups

Use these group names with scripts/run_ablation_matrix.py:

default split batch mbs lr sched slown mod smodel longpair lrctrl

longpair is the slow-update/meta-skill comparison-example ablation. It keeps all prompts and training settings unchanged and only overrides:

optimizer.longitudinal_pair_policy=changed
optimizer.longitudinal_pair_policy=unchanged

The default paper setting remains mixed.

lrctrl contains the two learning-rate-control baselines:

optimizer.lr_control_mode=autonomous
optimizer.lr_control_mode=none + optimizer.skill_update_mode=full_rewrite_minibatch

The autonomous run logs the chosen integer per step in lr_decision.json and lr_history.jsonl. The full-rewrite run removes the LR/edit-selection concept: each minibatch analyst produces a complete skill candidate, and aggregate/merge produces the candidate skill directly.

Batch-size values are:

8 / 24 / 40 / 56 / full

40 is the default point. full expands to the benchmark train size.

Launch Commands Used In This Session

The exact commands are recorded in launch_commands.sh and in docs/ablation_plan.md. The important current policy is:

SearchQA / SpreadsheetBench batch-only matrix can run at --max-parallel 8.
DocVQA matrix can run with its launcher at --max-parallel 8; later top-up used --max-parallel 16 only because completed runs were skipped and active roots were checked.
LiveMathBench is safe as API-only benchmark after the token cap fix.
ALFWorld must not be mixed into a 24-way run on this shared machine. Use --bench alfworld --max-parallel 1 only after memory is available.

Token And Timeout Fixes

LiveMathBench must use a large student completion cap:

max_completion_tokens=16384
timeout=300

The old 768/512 cap produced many empty visible responses because hidden reasoning consumed the budget.

ALFWorld must use:

max_completion_tokens=2048
empty response fallback -> <action>look</action>
missing action fallback -> <action>look</action>

Invalid Runs

Never fill paper tables from these archive directories:

outputs/ablation_livemath_alfworld_clean_20260503_155155_run/archive_livemath_token768_20260504_022258/
outputs/ablation_livemath_alfworld_clean_20260503_155155_run/archive_alfworld_token512_20260504_021417/
outputs/ablation_livemath_alfworld_clean_20260503_155155_run/archive_alfworld_empty_action_20260504_025311/
outputs/ablation_livemath_alfworld_clean_20260503_155155_run/archive_alfworld_prefallback_20260504_025402/
outputs/ablation_livemath_alfworld_clean_20260503_155155_run/archive_alfworld_oom_partial_20260504_050517/
outputs/ablation_livemath_alfworld_clean_20260503_155155_run/archive_alfworld_serial_lowmem_20260504_1300/

Current Resource Notes

ALFWorld model calls are API calls. The ablation branch now creates local ALFWorld/TextWorld environments through multiprocessing workers ported from skillopt_final_zzw, not through Ray actors. Old Ray-based archived runs are not valid for table fill. The observed historical failure mode in this session was system RAM pressure and Ray OOM prevention, not model GPU memory.

GPU memory currently shown by nvidia-smi came from unrelated Ray Serve visual models under:

/home/azureuser/workspace-gzy/zyf/gca-skill

Those processes are GroundingDINOModel / DA3Model, not the SkillReflection ablation ALFWorld run.

There are also unrelated ALFWorld jobs under:

/home/azureuser/zisu/skill_distill

Do not confuse those with this repository's ablation outputs.

Monitoring

Active run and duplicate output-root check:

$PY - <<'PY'
import subprocess, re, collections, time
try:
    raw = subprocess.check_output(["pgrep", "-af", "scripts/train.py"], text=True)
except subprocess.CalledProcessError:
    raw = ""
roots = []
for line in raw.splitlines():
    m = re.search(r"env\.out_root=([^\s]+)", line)
    if m:
        roots.append(m.group(1))
ctr = collections.Counter(roots)
print("time", time.strftime("%F %T"))
print("active_count", len(roots))
print("duplicates", [r.rsplit("/", 1)[-1] for r, c in ctr.items() if c > 1])
for root in sorted(roots):
    print(root.rsplit("/", 1)[-1])
PY

Error scan:

rg -n "Traceback|ERROR|Error code|AuthenticationError|BadRequest|RateLimit|content_filter|Killed|OutOfMemory|CUDA out of memory|\\[FAIL\\]|LLM call failed" \
  outputs/ablation_docvqa_20260503_160225_run/logs \
  outputs/ablation_livemath_alfworld_clean_20260503_155155_run/logs \
  outputs/ablation_batch_searchqa_spreadsheet_20260503_153902_run/logs \
  -g '*.log' | tail -100 || true

Resource checks:

free -h | sed -n '1,3p'
df -h /tmp
du -sh /tmp/ray 2>/dev/null || true
nvidia-smi

Filling Tables

Only use top-level summary.json from valid run roots. Fill docs/ablation_paper_tables.md from:

best_selection_hard
baseline_test_hard
test_hard
test_delta_hard
token_summary._total.total_tokens