---
license: mit
tags:
- nlp
- math
language:
- en
pipeline_tag: text-generation
---

## Rho-1: Not All Tokens Are What You Need


The Rho-1 series are pretrained language models that utilize Selective Language Modeling (SLM) objectives.
In math reasoning pretraining, SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.


For more details please check our [github](https://github.com/microsoft/rho) and [paper](https://arxiv.org/abs/2404.07965).