readme: add some interesting details about umT5

README.md

@@ -4,6 +4,23 @@ license: mit
 
 # umT5 Small
 
+The UMT5 model was proposed in [UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining](https://openreview.net/forum?id=kXwdL1cWOAi)
+by Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, Noah Constant.
+
+The abstract from the paper is the following:
+
+*Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance
+between different languages. However previous work has not systematically evaluated the efficacy of different
+pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax,
+that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly
+capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a
+range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax
+outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our
+contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters
+across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.*
+
+# Integration into Transformers
+
 Overview of umT5 model integration:
 
 * Transformers Integration is on-going, see this awesome [PR](https://github.com/huggingface/transformers/pull/22626) by @agemagician!