Procedural Math Reasoning Robustness: dataset for "From Recognition to Reconstruction: Towards Robust Procedural Mathematical Reasoning"
Description
A procedural-robustness panel of 2,800 mathematical statements (1,400 equivalent reformulations of named theorems and 1,400 constructed contrasts) with reference theorem labels and ordered transformation sequences, 44,800 scored outputs from four open-weight models under four conditions (closed-book and retrieval-augmented), the retrieval corpora (306 theorem cards, 17 operation cards) and the full protocol (prompts, output schema, operation vocabulary and run settings).
Dataset accompanying the paper From Recognition to Reconstruction: Towards Robust Procedural Mathematical Reasoning, 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026), EMNLP 2026 (https://www.carlostoxtli.com/papers/mathnlp2026-from-recognition-to-reconstruction.pdf).
Paper abstract. Mathematical reasoning systems should be consistent when the same concept is expressed in equivalent forms. This requires not only recognizing shared mathematical meaning but also identifying the representation changes and reconstructing a traceable route between forms. We study this problem through mathematical theorems, whose equivalent formulations preserve the same underlying result while differing substantially in representation. In this setting, theorem recognition is only the first step: a robust system must also identify the intervening transformations and recover their order. We introduce an evaluation of robust procedural mathematical reasoning with three components: theorem identification, transformation identification, and ordered reconstruction. Using validated theorem–representation pairs from an existing corpus, we build a procedural-robustness dataset with typed operations, reference procedures, and matched contrasts that distinguish valid reformulations from invalid ones. We first measure closed-book performance of four open-weight models to establish a baseline, then evaluate retrieval-augmented generation (RAG) as support mechanism for the same task. Models identify theorem identity more reliably than they recover the transformations and ordered routes connecting equivalent representations. RAG improves every stage, but procedural reconstruction remains the main bottleneck. The framework supports more consistent and traceable mathematical systems, with potential applications in mathematical search, tutoring, autoformalization, and scientific discovery.
Known limitations.
- Reference routes come from corpus metadata, not machine-checked proofs.
- Contrasts were scored as negatives; their invalidity has not been independently established.
- Results are unchanged from the original experiment, including 1,045 invalid outputs; raw generations and retry logs are not included.
License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/). Theorem statements derive from the TREAT corpus, which is built from English Wikipedia (CC BY-SA 4.0); share-alike therefore applies.
Files
croissant.json
Files
(1.0 MB)
| Name | Size | |
|---|---|---|
|
md5:d0be1a423c2037378eac87470b94f953
|
1.6 kB | Download |
|
md5:64e3a71feecca8c6d6e6cce316a1ad95
|
27.3 kB | Preview Download |
|
md5:dc001a577eb75b24d8bf5d4afd228523
|
591 Bytes | Download |
|
md5:e07d7e546da8344872445fa95eba20c2
|
984.5 kB | Preview Download |
|
md5:2436da323f73082fada19f6918a05053
|
8.7 kB | Preview Download |
Additional details
Related works
- Is identical to
- Dataset: https://huggingface.co/datasets/ai4he/procedural-math-reasoning-robustness (URL)
- Is supplement to
- Conference paper: https://www.carlostoxtli.com/papers/mathnlp2026-from-recognition-to-reconstruction.pdf (URL)