Update README.md

README.md

@@ -9,199 +9,468 @@ metrics:
 - accuracy
 - f1
 - perplexity
+pipeline_tag: text-generation
 ---
 
-# Model Card for Model ID
+# **Model Card for Basque Llama 13b**
 
-<!-- Provide a quick summary of what the model is/does. -->
+Basque LLaMA is a collection of foundation models specifically tuned for Basque. Based on Meta’s LLaMA 2 model family, these models were further trained with Euscrawl, a highly curated Basque corpora ([Artetxe et al., 2022](https://aclanthology.org/2022.emnlp-main.499/)). Ranging from 7 billion to 70 billion parameters, these models are currently the biggest and best-performing LLMs built for Basque. This is the 13b repository, links to other models can be found in the index at the bottom.
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
 
-## Model Details
+# **Model Details**
 
-### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+## **Model Description**
 
+Basque LLaMA is a family of Large Language Models (LLM) based on Meta’s [LLaMA models](https://huggingface.co/meta-llama). Current LLMs exhibit incredible performance for high-resource languages such as English, but, in the case of Basque and other low-resource languages, their performance is close to a random guesser. These limitations widen the gap between high- and low-resource languages when it comes to digital development. We present Basque LLaMA to overcome these limitations and promote the development of LLM-based technology and research for the Basque language. Basque LLaMA models follow the same architecture as their original counterparts and were further trained in Euscrawl v1 ([Artetxe et al., 2022](https://aclanthology.org/2022.emnlp-main.499/)), a high-quality Basque corpora. 
 
+The models are released in three sizes: 7B, 13B and 70B.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
 
-### Model Sources [optional]
 
-<!-- Provide the basic links for the model. -->
+* **Developed by:** HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
+* **Model type:** Language model
+* **Language(s) (NLP):** en, eu
+* **License:** llama2
+* **Parent Model:** meta-llama/Llama-2-13b
+* **Contact:** [email protected] 
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
 
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
+## **Getting started**
 
 Use the code below to get started with the model.
 
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure 
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
+```python
 
-<!-- This should link to a Dataset Card if possible. -->
+from transformers import pipeline
 
-[More Information Needed]
+pipe = pipeline("text-generation", model=”HiTZ/basque-llama-2-7b-v1”)
 
-#### Factors
+text = "Euskara adimen artifizialera iritsi da!"
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+pipe(text, max_new_tokens=50, num_beams=5)
 
-[More Information Needed]
+>> [
+ {
+  'generated_text': 'Euskara adimen artifizialera iritsi da!\nEuskararen eta adimen artifizialaren arteko harremana aspaldikoa da,'
+  ' baina azken urteotan aurrerapauso handiak eman dira arlo horretan'
+ }
+]
 
-#### Metrics
+```
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
+# **Uses**
 
-### Results
+Basque LLaMA models are intended to be used with Basque data; for any other language the performance is not guaranteed. Same as the original, Basque LLaMA inherits the [LLaMA-2 License](https://ai.meta.com/llama/license/) which allows for commercial and research use. 
 
-[More Information Needed]
 
-#### Summary
+## **Direct Use**
 
+Basque LLaMA family models are pre-trained LLMs without any task-specific or instruction fine-tuning. That is, the model can either be prompted to perform a specific task or further fine-tuned for specific use cases.
 
 
-## Model Examination [optional]
+## **Out-of-Scope Use**
 
-<!-- Relevant interpretability work for the model goes here -->
+The model was not fine-tuned to follow instructions or to work as a chat assistant, therefore, this kind of usage is not tested nor recommended. 
 
-[More Information Needed]
 
-## Environmental Impact
+# **Bias, Risks, and Limitations**
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+In an effort to alleviate the potentially disturbing or harmful content, Basque LLaMA has been trained on carefully selected and processed data which comes mainly from local media, national/regional newspapers, encyclopedias and blogs (see Euscrawl below). Still, the model is based on LLaMA models and can potentially carry the same bias, risk and limitations. 
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+Please see the LLaMA’s _Ethical Considerations and Limitations _for further information.
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
+# **Training Details**
 
-### Model Architecture and Objective
 
-[More Information Needed]
+## **Training Data**
 
-### Compute Infrastructure
+The models were trained on EusCrawl v1, a high-quality corpus for Basque comprising 1.72M documents, 288M words, totalling 2.1GiB of uncompressed text. EusCrawl was built using ad-hoc scrapers to extract text from 33 Basque websites with high-quality content, resulting in cleaner text compared to general-purpose approaches.	
 
-[More Information Needed]
+See more details in the [EusCrawl](https://huggingface.co/datasets/HiTZ/euscrawl) dataset card. 
 
-#### Hardware
+Additionally, 100K documents of English data randomly selected from the [Pile](https://huggingface.co/datasets/EleutherAI/pile) dataset were also included to avoid catastrophic forgetting.
 
-[More Information Needed]
 
-#### Software
+## **Training Procedure**
 
-[More Information Needed]
+The models were trained using the GPT-Neox library on the HPC CINECA computing cluster. All the models were approximately trained with an effective batch size of 2M tokens for 1000 to 2000 steps. 
 
-## Citation [optional]
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+<table>
+  <tr>
+   <td>Model
+   </td>
+   <td>Steps
+   </td>
+   <td>Sequence length
+   </td>
+   <td>Effective Batch size
+   </td>
+   <td>Total tokens
+   </td>
+   <td>GPU hours
+   </td>
+  </tr>
+  <tr>
+   <td>Basque LLaMA 7B
+   </td>
+   <td><p style="text-align: right">
+2000</p>
 
-**BibTeX:**
+   </td>
+   <td><p style="text-align: right">
+4096</p>
 
-[More Information Needed]
+   </td>
+   <td><p style="text-align: right">
+2M tokens/step</p>
 
-**APA:**
+   </td>
+   <td><p style="text-align: right">
+4B</p>
 
-[More Information Needed]
+   </td>
+   <td><p style="text-align: right">
+359.2h</p>
 
-## Glossary [optional]
+   </td>
+  </tr>
+  <tr>
+   <td>Basque LLaMA 13B
+   </td>
+   <td><p style="text-align: right">
+1000</p>
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+   </td>
+   <td><p style="text-align: right">
+4096</p>
 
-[More Information Needed]
+   </td>
+   <td><p style="text-align: right">
+2M tokens/step</p>
 
-## More Information [optional]
+   </td>
+   <td><p style="text-align: right">
+2B</p>
 
-[More Information Needed]
+   </td>
+   <td><p style="text-align: right">
+468.8h</p>
 
-## Model Card Authors [optional]
+   </td>
+  </tr>
+  <tr>
+   <td>Basque LLaMA 70B
+   </td>
+   <td><p style="text-align: right">
+1680</p>
 
-[More Information Needed]
+   </td>
+   <td><p style="text-align: right">
+4096</p>
 
-## Model Card Contact
+   </td>
+   <td><p style="text-align: right">
+2M tokens/step</p>
 
-[More Information Needed]
+   </td>
+   <td><p style="text-align: right">
+3.4B</p>
+
+   </td>
+   <td><p style="text-align: right">
+*6475.52h</p>
+
+   </td>
+  </tr>
+</table>
+
+
+* indicates the time for the entire training process (2000 steps), however the weights of the step 1680 are shared as it is the best checkpoint according to validation loss.
+
+
+# **Evaluation**
+
+We evaluated the models on zero-shot and few-shot settings on generative, multiple-choice and classification tasks. We used the basque partitions of each dataset.
+
+
+## **Testing Data, Factors & Metrics**
+
+
+### **Testing Data**
+
+
+
+* **Belebele** ([Bandarkar et al.](https://arxiv.org/abs/2308.16884)): Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. We evaluated the model in a 5-shot fashion.
+    * Data card: [https://huggingface.co/datasets/facebook/belebele](https://huggingface.co/datasets/facebook/belebele)
+* **X-StoryCloze**: XStoryCloze consists of the professionally translated version of the English StoryCloze dataset to 10 non-English languages. Story Cloze is a new commonsense reasoning dataset which consists of choosing the correct ending to a four-sentence story. We evaluated the model in a 0-shot fashion.
+    * Data card: [https://huggingface.co/datasets/juletxara/xstory_cloze](https://huggingface.co/datasets/juletxara/xstory_cloze)
+* **BasqueGLUE** ([Urbizu et al.](https://aclanthology.org/2022.lrec-1.172.pdf)): BasqueGLUE is a NLU benchmark for Basque. We evaluated the model in a 5-shot fashion on the following tasks:
+    * Data card:[ https://huggingface.co/datasets/orai-nlp/basqueGLUE](https://huggingface.co/datasets/orai-nlp/basqueGLUE).
+    * Tasks:
+        * **BEC2016eu**: Sentiment analysis on tweets about the 2016 Basque elections campaign.
+        * **VaxxStance**: Stance detection on tweets around the anti-vaccine movement.
+        * **BTHCv2**: Topic classification of news extracts with 12 categories.
+        * **EpecKorrefBin**: Correference detection task similar to WSC.
+        * **QNLIeu**: Q&A NLI built from the Basque Wikipedia.
+        * **WiCeu**: Basque Word-in-Context task.
+
+
+### **Metrics**
+
+
+
+* **Accuracy**: Belebele, X-StoryCloze, EpecKorrefBin, QNLI-eu, and, WiC-eu
+* **Micro F1**: BEC2016-eu and BHTCv2
+* **Macro F1**: VaxxStance (favor & against)
+
+
+## **Results**
+
+The model was evaluated using the LM Evaluation harness library from Eleuther AI. In order to reproduce our results please refer to our [fork](https://github.com/naiarapm/lm-evaluation-harness/tree/basqueglue) that includes the implementation for the mentioned datasets.
+
+
+<table>
+  <tr>
+   <td><strong>Model</strong>
+   </td>
+   <td><strong>Belebele</strong>
+   </td>
+   <td><strong>X-StoryCloze</strong>
+   </td>
+   <td><strong>BEC</strong>
+   </td>
+   <td><strong>Vaxx</strong>
+   </td>
+   <td><strong>BHTC</strong>
+   </td>
+   <td><strong>coref</strong>
+   </td>
+   <td><strong>QNLI</strong>
+   </td>
+   <td><strong>WiC</strong>
+   </td>
+   <td><strong>Average</strong>
+   </td>
+  </tr>
+  <tr>
+   <td>Random
+   </td>
+   <td>25.00
+   </td>
+   <td>50.00
+   </td>
+   <td>33.33
+   </td>
+   <td>33.33
+   </td>
+   <td>8.33
+   </td>
+   <td>50.00
+   </td>
+   <td>50.00
+   </td>
+   <td>50.00
+   </td>
+   <td>37.50
+   </td>
+  </tr>
+  <tr>
+   <td>LLaMA 2 7B
+   </td>
+   <td>26.22
+   </td>
+   <td>50.43
+   </td>
+   <td>41.63
+   </td>
+   <td>18.60
+   </td>
+   <td>20.06
+   </td>
+   <td>50.94
+   </td>
+   <td>48.32
+   </td>
+   <td>49.64
+   </td>
+   <td>38.23
+   </td>
+  </tr>
+  <tr>
+   <td>LLaMA 2 13B
+   </td>
+   <td>32.00
+   </td>
+   <td>50.63
+   </td>
+   <td>41.09
+   </td>
+   <td>18.25
+   </td>
+   <td>27.35
+   </td>
+   <td>49.23
+   </td>
+   <td>48.74
+   </td>
+   <td>49.21
+   </td>
+   <td>39.56
+   </td>
+  </tr>
+  <tr>
+   <td>LLaMA 2 70B
+   </td>
+   <td>33.56
+   </td>
+   <td>51.62
+   </td>
+   <td>47.47
+   </td>
+   <td>21.01
+   </td>
+   <td>31.01
+   </td>
+   <td>52.98
+   </td>
+   <td>51.26
+   </td>
+   <td>51.57
+   </td>
+   <td>42.56
+   </td>
+  </tr>
+  <tr>
+   <td>BLOOM 7B
+   </td>
+   <td>27.00
+   </td>
+   <td>57.18
+   </td>
+   <td>37.94
+   </td>
+   <td>20.72
+   </td>
+   <td>39.10
+   </td>
+   <td>48.21
+   </td>
+   <td>47.48
+   </td>
+   <td>47.57
+   </td>
+   <td>40.65
+   </td>
+  </tr>
+  <tr>
+   <td>XGLM 7B
+   </td>
+   <td>23.88
+   </td>
+   <td>57.71
+   </td>
+   <td>39.94
+   </td>
+   <td>21.58
+   </td>
+   <td>36.73
+   </td>
+   <td>50.94
+   </td>
+   <td>50.42
+   </td>
+   <td>49.21
+   </td>
+   <td>41.30
+   </td>
+  </tr>
+  <tr>
+   <td><strong>Basque LLaMA 7B</strong>
+   </td>
+   <td>35.67
+   </td>
+   <td>63.13
+   </td>
+   <td>55.61
+   </td>
+   <td>45.93
+   </td>
+   <td>44.44
+   </td>
+   <td>50.43
+   </td>
+   <td>55.04
+   </td>
+   <td>50.14
+   </td>
+   <td>50.05
+   </td>
+  </tr>
+  <tr>
+   <td><strong>Basque LLaMA 13B</strong>
+   </td>
+   <td>53.56
+   </td>
+   <td>65.85
+   </td>
+   <td>53.23
+   </td>
+   <td>48.66
+   </td>
+   <td><strong>53.61</strong>
+   </td>
+   <td>62.52
+   </td>
+   <td>57.14
+   </td>
+   <td>54.21
+   </td>
+   <td>56.10
+   </td>
+  </tr>
+  <tr>
+   <td><strong>Basque LLaMA 70B</strong>
+   </td>
+   <td><strong>71.78</strong>
+   </td>
+   <td><strong>67.57</strong>
+   </td>
+   <td><strong>63.52</strong>
+   </td>
+   <td><strong>48.95</strong>
+   </td>
+   <td>49.51
+   </td>
+   <td><strong>79.90</strong>
+   </td>
+   <td><strong>58.82</strong>
+   </td>
+   <td><strong>55.50</strong>
+   </td>
+   <td><strong>61.94</strong>
+   </td>
+  </tr>
+</table>
+
+
+
+# **Environmental Impact**
+
+Carbon emissions are estimated using the[ Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in[ Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+
+
+* **Hardware Type:** HPC Cluster, 4x A100 64Gb nodes
+* **Hours used:** 359.2h + 468.8h + 6475.52h = 7303.52h
+* **Compute cluster:** CINECA HPC
+* **Compute Region:** Italy
+* **Carbon Emitted:** 673.75kg CO<sub>2</sub> eq
+
+
+# **Acknowledgements**
+
+This work has been partially supported by the Basque Government (IKER-GAITU project). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2023E01-013.