Model Card

Some additional information for the model card, based on the format we are using as part of our effort to standardise model cards
Feel free to merge if you are ok with the changes! (cc

@Marissa



@Meg

)

README.md

@@ -7,15 +7,58 @@ datasets:
 
 # CamemBERT: a Tasty French Language Model
 
-## Introduction
+## Table of Contents
+- [Model Details](#model-details)
+- [Uses](#uses)
+- [Risks, Limitations and Biases](#risks-limitations-and-biases)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Citation Information](#citation-information)
+- [How to Get Started With the Model](#how-to-get-started-with-the-model)
+
+
+## Model Details
+- **Model Description:**
+CamemBERT is a state-of-the-art language model for French based on the RoBERTa model.
+It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.
+- **Developed by:**  Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
+- **Model Type:** Fill-Mask
+- **Language(s):** French
+- **License:** MIT
+- **Parent Model:** See the [RoBERTa base model](https://huggingface.co/roberta-base) for more information about the RoBERTa   base model.
+- **Resources for more information:**
+	- [Research Paper](https://arxiv.org/abs/1911.03894)
+  - [Camembert Website](https://camembert-model.fr/)
+  
+  
+## Uses
 
-[CamemBERT](https://arxiv.org/abs/1911.03894) is a state-of-the-art language model for French based on the RoBERTa model.
+#### Direct Use
 
-It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.
+This model can be used for Fill-Mask tasks.
+
+
+## Risks, Limitations and Biases
+**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
+
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
+
+This model was pretrinaed on a subcorpus of OSCAR multilingual corpus. Some of the limitations and risks associated with the OSCAR dataset, which are further detailed in the [OSCAR dataset card](https://huggingface.co/datasets/oscar), include the following: 
+
+> The quality of some OSCAR sub-corpora might be lower than expected, specifically for the lowest-resource languages.
+
+> Constructed from Common Crawl, Personal and sensitive information might be present.
+
+
+
+## Training
 
-For further information or requests, please go to [Camembert Website](https://camembert-model.fr/)
 
-## Pre-trained models
+#### Training Data
+OSCAR or Open Super-large Crawled Aggregated coRpus is a multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the Ungoliant architecture.
+
+
+#### Training Procedure
 
 | Model                          | #params                        | Arch. | Training data                     |
 |--------------------------------|--------------------------------|-------|-----------------------------------|
@@ -26,7 +69,25 @@ For further information or requests, please go to [Camembert Website](https://ca
 | `camembert/camembert-base-oscar-4gb`     | 110M    | Base  | Subsample of OSCAR (4 GB of text) |
 | `camembert/camembert-base-ccnet-4gb`     | 110M    | Base  | Subsample of CCNet (4 GB of text) |
 
-## How to use CamemBERT with HuggingFace
+## Evaluation
+
+
+The model developers evaluated CamemBERT using four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI).
+
+
+
+## Citation Information
+
+```bibtex
+@inproceedings{martin2020camembert,
+  title={CamemBERT: a Tasty French Language Model},
+  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
+  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
+  year={2020}
+}
+```
+
+## How to Get Started With the Model
 
 ##### Load CamemBERT and its sub-word tokenizer :
 ```python
@@ -95,21 +156,3 @@ all_layer_embeddings[5]
 #         ...,
 ```
 
-
-## Authors 
-
-CamemBERT was trained and evaluated by Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
-
-
-## Citation
-If you use our work, please cite:
-
-```bibtex
-@inproceedings{martin2020camembert,
-  title={CamemBERT: a Tasty French Language Model},
-  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
-  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
-  year={2020}
-}
-```
-