Model Card (#1)

- Model Card (aca67c938dda9a8e033550315b764aa6c42be1f9)


Co-authored-by: Ezi Ozoani <[email protected]>

README.md

@@ -7,15 +7,58 @@ datasets:
 
 # CamemBERT: a Tasty French Language Model
 
-## Introduction
+## Table of Contents
+- [Model Details](#model-details)
+- [Uses](#uses)
+- [Risks, Limitations and Biases](#risks-limitations-and-biases)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Citation Information](#citation-information)
+- [How to Get Started With the Model](#how-to-get-started-with-the-model)
+
+
+## Model Details
+- **Model Description:**
+CamemBERT is a state-of-the-art language model for French based on the RoBERTa model.
+It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.
+- **Developed by:**  Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
+- **Model Type:** Fill-Mask
+- **Language(s):** French
+- **License:** MIT
+- **Parent Model:** See the [RoBERTa base model](https://huggingface.co/roberta-base) for more information about the RoBERTa   base model.
+- **Resources for more information:**
+	- [Research Paper](https://arxiv.org/abs/1911.03894)
+  - [Camembert Website](https://camembert-model.fr/)
+  
+  
+## Uses
 
-[CamemBERT](https://arxiv.org/abs/1911.03894) is a state-of-the-art language model for French based on the RoBERTa model.
+#### Direct Use
 
-It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.
+This model can be used for Fill-Mask tasks.
+
+
+## Risks, Limitations and Biases
+**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
+
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
+
+This model was pretrinaed on a subcorpus of OSCAR multilingual corpus. Some of the limitations and risks associated with the OSCAR dataset, which are further detailed in the [OSCAR dataset card](https://huggingface.co/datasets/oscar), include the following: 
+
+> The quality of some OSCAR sub-corpora might be lower than expected, specifically for the lowest-resource languages.
+
+> Constructed from Common Crawl, Personal and sensitive information might be present.
+
+
+
+## Training
 
-For further information or requests, please go to [Camembert Website](https://camembert-model.fr/)
 
-## Pre-trained models
+#### Training Data
+OSCAR or Open Super-large Crawled Aggregated coRpus is a multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the Ungoliant architecture.
+
+
+#### Training Procedure
 
 | Model                          | #params                        | Arch. | Training data                     |
 |--------------------------------|--------------------------------|-------|-----------------------------------|
@@ -26,7 +69,25 @@ For further information or requests, please go to [Camembert Website](https://ca
 | `camembert/camembert-base-oscar-4gb`     | 110M    | Base  | Subsample of OSCAR (4 GB of text) |
 | `camembert/camembert-base-ccnet-4gb`     | 110M    | Base  | Subsample of CCNet (4 GB of text) |
 
-## How to use CamemBERT with HuggingFace
+## Evaluation
+
+
+The model developers evaluated CamemBERT using four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI).
+
+
+
+## Citation Information
+
+```bibtex
+@inproceedings{martin2020camembert,
+  title={CamemBERT: a Tasty French Language Model},
+  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
+  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
+  year={2020}
+}
+```
+
+## How to Get Started With the Model
 
 ##### Load CamemBERT and its sub-word tokenizer :
 ```python
@@ -95,21 +156,3 @@ all_layer_embeddings[5]
 #         ...,
 ```
 
-
-## Authors 
-
-CamemBERT was trained and evaluated by Louis Martin\*, Benjamin Muller\*, Pedro Javier Ortiz Suárez\*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
-
-
-## Citation
-If you use our work, please cite:
-
-```bibtex
-@inproceedings{martin2020camembert,
-  title={CamemBERT: a Tasty French Language Model},
-  author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
-  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
-  year={2020}
-}
-```
-