model documentation

README.md

@@ -1,19 +1,185 @@
+
 ---
-language: "cs"
+language: cs
+license: cc-by-nc-sa-4.0
 tags:
 - Czech
 - RoBERTa
 - ÚFAL
-license: "cc-by-nc-sa-4.0"
 ---
 
-# RobeCzech model
-
+# Model Card for RobeCzech
+# Model Details
+ 
+## Model Description
+ 
 RobeCzech is a monolingual RoBERTa language representation model trained on Czech data.
+ 
+- **Developed by:** Institute of Formal and Applied Linguistics, Charles University, Prague (UFAL)
+- **Shared by [Optional]:** Hugging Face and [LINDAT](https://hdl.handle.net/11234/1-3691)
+- **Model type:** Fill-Mask
+- **Language(s) (NLP):** cs
+- **License:** cc-by-nc-sa-4.0
+- **Related Models:** More information needed
+  - **Parent Model:** RoBERTa
+- **Resources for more information:** 
+  - [Associated Paper](https://arxiv.org/abs/2105.11314)
+ 
+# Uses
+ 
+## Direct Use
+ 
+Fill-Mask tasks.
+ 
+## Downstream Use [Optional]
+ 
+Morphological tagging and lemmatization, dependency parsing, named entity recognition and semantic parsing.
+ 
+ 
+## Out-of-Scope Use
+ 
+ 
+More information needed
+ 
+ 
+# Bias, Risks, and Limitations
+ 
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
+ 
+ 
+## Recommendations
+ 
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+ 
+ 
+# Training Details
+ 
+## Training Data
+ 
+The model creators note in the [associated paper](https://arxiv.org/pdf/2105.11314.pdf):
+> We trained RobeCzech on a collection of the following publicly available texts:
+    -   SYN v4, a large corpus of contemporary written Czech, 4,188M tokens; 
+    -   Czes, a collection of Czech newspaper and magazine articles, 432M tokens; 
+    -     documents with at least 400 tokens from the Czech part of the web corpus.W2C , tokenized with MorphoDiTa, 16M tokens;
+    -     plain texts extracted from Czech Wikipedia dump 20201020 using WikiEx-tractor, tokenized with MorphoDiTa, 123M tokens
 
-RobeCzech model is released publicly at [LINDAT](https://hdl.handle.net/11234/1-3691) and [Hugging Face](https://huggingface.co/ufal/robeczech-base).
+> All these corpora contain whole documents, even if the SYN v4 is block-shuffled (blocks with at most 100 words respecting sentence boundaries are permuted in a document) and in total contain 4,917M tokens.
+ 
+ 
+ 
+## Training Procedure
+ 
+### Preprocessing
+ 
+The texts are tokenized into subwords with a byte-level BPE (BBPE) to- kenizer [33]. The tokenizer is trained on the entire corpus and we limit its vocabulary size to 52,000 items.
+ 
+### Speeds, Sizes, Times
+The model creators note in the [associated paper](https://arxiv.org/pdf/2105.11314.pdf):
+> The training batch size is 8,192 and each training batch consists of sentences sampled contiguously, even across document boundaries, such that the total length of each sample is at most 512 tokens (FULL-SENTENCES setting). We use Adam optimizer with β1 = 0.9 and β2 = 0.98 to minimize the masked language-modeling objective. 
+ 
+ 
+ 
+# Evaluation
+ 
+## Testing Data, Factors & Metrics
+ 
+### Testing Data
+The model creators note in the [associated paper](https://arxiv.org/pdf/2105.11314.pdf):
+> We evaluate RobeCzech in five NLP tasks, three of them leveraging frozen contextualized word embeddings, two approached with fine-tuning:
+morphological analysis and lemmatization: frozen contextualized word embeddings,
+dependency parsing: frozen contextualized word embeddings,
+  named entity recognition: frozen contextualized word embeddings,
+ semantic parsing: fine-tuned,
+  sentiment analysis: fine-tuned.
+ 
+ 
+ 
+### Factors
+ 
+More information needed
+ 
+### Metrics
+ 
+Morphosynt PDT3.5 (POS) | Morphosynt PDT3. (LAS) | Morphosynt UD2.3 (XPOS) | Morphosynt UD2.3 ( LAS) | 
+NER CNEC1.1 (nested) | NER CNEC1.1  (flat) | Semant. PTG (Avg) | Sentim. CDF (F1) |
+ 
+ 
+## Results 
+ 
+| Model     | Morphosynt PDT3.5   (POS) (LAS) | Morphosynt UD2.3  (XPOS) (LAS) | NER CNEC1.1  (nested) (flat) | Semant. PTG  (Avg) (F1) |
+|-----------|---------------------------------|--------------------------------|------------------------------|-------------------------|
+| RobeCzech | 98.50      91.42                | 98.31    93.77                 | 87.82	           87.47        | 92.36           80.13   |
+ 
+ 
+# Model Examination
+ 
+More information needed
+ 
+# Environmental Impact
+ 
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+ 
+- **Hardware Type:** 8 QUADRO P5000 GPU 
+- **Hours used:** 2190 (~3 months)
+- **Cloud Provider:** More information needed
+- **Compute Region:** More information needed
+- **Carbon Emitted:** More information needed
+ 
+# Technical Specifications [optional]
+ 
+## Model Architecture and Objective
+The model creators note in the [associated paper](https://arxiv.org/pdf/2105.11314.pdf):
+We employ the standard text classification architecture consisting of a BERT encoder, followed by a softmax-activated classification layer processing the computed embedding of the given document text obtained from the CLS token embedding from the last layer
+ 
+ 
+## Compute Infrastructure
+ 
+### Hardware
+ 
+8 QUADRO P5000 GPU 
+ 
+### Software
+ 
+More information needed
+ 
+# Citation
+ 
+**APA:**
+ 
+```
+Straka, M., Náplava, J., Straková, J., & Samuel, D. (2021). RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model. arXiv. https://doi.org/10.1007/978-3-030-83527-9_17
+```
+ 
+ 
+# Glossary [optional]
+ 
+More information needed
+ 
+# More Information [optional]
+ 
+More information needed
+ 
+# Model Card Authors [optional]
+ 
+ Institute of Formal and Applied Linguistics, Charles University, Prague (UFAL), in collaboration with Ezi Ozoani and the Hugging Face Team.
+ 
+# Model Card Contact
+ 
+More information needed
+ 
+# How to Get Started with the Model
+ 
+Use the code below to get started with the model.
+ 
+<details>
+<summary> Click to expand </summary>
 
-Please cite the corresponding publication:
-- Milan Straka, Jakub Náplava, Jana Straková and David Samuel: Czech RoBERTa, a monolingual contextualized language representation model. Accepted to TSD 2021.
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+ 
+tokenizer = AutoTokenizer.from_pretrained("ufal/robeczech-base")
+ 
+model = AutoModelForMaskedLM.from_pretrained("ufal/robeczech-base")
+ ```
+</details>
 
-Preprint of the paper is available at https://arxiv.org/abs/2105.11314.