Mistral-7B-ProXMath

ArXiv | Data: OpenWebMath-Pro | Code

Mistral-7B-ProXMath is a math-adapted Mistral-7B-v0.1 model that is continually pre-trained on OpenWebMath-Pro (a refined version by ProX) for 10B tokens.

Evaluations

ProX models are evaluated on 9 common math reasoning benchmarks.

Model	asdiv	gsm8k	mathqa	mawps	minerva_math	mmlu_stem	sat_math	svamp	tabmwp	average
Mistral-7B-v0.1	68.5	40.6	32.3	87.0	11.4	50.0	56.2	65.4	52.9	51.6
Mistral-7B-ProXMath	72.9	51.0	53.0	89.2	22.4	54.2	75.0	64.9	49.8	59.2

Citation

@article{zhou2024programming,
  title={Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale},
  author={Zhou, Fan and Wang, Zengzhi and Liu, Qian and Li, Junlong and Liu, Pengfei},
  journal={arXiv preprint arXiv:2409.17115},
  year={2024}
}