github/microsoft-SkillOpt
1
0
Fork 0
mirror of https://github.com/microsoft/SkillOpt.git synced 2026-08-03 07:02:46 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix: sync LiveMath split metadata

Browse Source 
Fixes #158
This commit is contained in:
Matt Van Horn
2026-07-26 03:00:47 -07:00
parent ed0d13635b
commit cd63ada183
4 changed files with 31 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.osc-metadata/sync.json Normal file
View File

@@ -0,0 +1,5 @@
{
"fork_synced_at": "2026-07-26T09:38:34.869801+00:00",
"commits_behind_before_sync": 210,
"action_taken": "synced"
}

2
data/README.md
View File

@@ -29,7 +29,7 @@ Each `items.json` contains only stable IDs or source-path hints.
| Manifest directory | Benchmark | Counts | Coverage | Raw data source | `split_dir` |
|---|---|---:|---|---|---|
| `searchqa_id_split/` | SearchQA | 400 / 200 / 1400 | Official HF dataset IDs | [lucadiliello/searchqa](https://huggingface.co/datasets/lucadiliello/searchqa) | `data/searchqa_split` |
| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 18 / 124 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 17 / 125 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
| `docvqa_id_split/` | DocVQA | 107 / 53 / 374 | 10% subset of validation | [lmms-lab/DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA) | `data/docvqa/splits` |
| `officeqa_id_split/` | OfficeQA | 50 / 24 / 172 | OfficeQA Full | [databricks/officeqa](https://huggingface.co/datasets/databricks/officeqa) | `data/officeqa_split` |
| `spreadsheetbench_id_split/` | SpreadsheetBench | 80 / 40 / 280 | SpreadsheetBench Verified 400 | [KAKA22/SpreadsheetBench](https://huggingface.co/datasets/KAKA22/SpreadsheetBench) | `data/spreadsheetbench_split` |

4
data/livemathematicianbench_id_split/split_manifest.json
View File

@@ -16,8 +16,8 @@
"split_seed": 42,
"counts": {
"train": 35,
"val": 18,
"test": 124
"val": 17,
"test": 125
},
"item_fields": [
"id",

23
tests/test_data_manifests.py Normal file
View File

@@ -0,0 +1,23 @@
import json
from pathlib import Path
from skillopt.datasets.base import SPLIT_NAMES
DATA_DIR = Path(__file__).resolve().parents[1] / "data"
def test_split_manifest_counts_match_item_files() -> None:
manifest_paths = sorted(DATA_DIR.glob("*/split_manifest.json"))
assert manifest_paths, f"No split manifests found under {DATA_DIR}"
for manifest_path in manifest_paths:
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
for split_name in SPLIT_NAMES:
declared_count = manifest["counts"][split_name]
items_path = manifest_path.parent / split_name / "items.json"
actual_count = len(json.loads(items_path.read_text(encoding="utf-8")))
assert declared_count == actual_count, (
f"{manifest_path} split {split_name!r}: declared count {declared_count}, actual count {actual_count}"
)