---
language: 
  - multilingual
  - af
  - am
  - ar
  - az
  - be
  - bg
  - bn
  - ca
  - cs
  - cy
  - da
  - de
  - el
  - en
  - eo
  - es
  - et
  - eu
  - fa
  - fi
  - fr
  - ga
  - gl
  - gu
  - ha
  - he
  - hi
  - hr
  - hu
  - hy
  - id
  - is
  - it
  - ja
  - ka
  - kk
  - km
  - kn
  - ko
  - ku
  - ky
  - la
  - lo
  - lt
  - lv
  - mk
  - ml
  - mn
  - mr
  - ms
  - my
  - ne
  - nl
  - no
  - or
  - pa
  - pl
  - ps
  - pt
  - ro
  - ru
  - sa
  - si
  - sk
  - sl
  - so
  - sq
  - sr
  - sv
  - sw
  - ta
  - te
  - th
  - tl
  - tr
  - uk
  - ur
  - uz
  - vi
  - zh
license: mit
---

# xmod-base

X-MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](http://dx.doi.org/10.18653/v1/2022.naacl-main.255) (Pfeiffer et al., NAACL 2022) and first released in [this repository](https://github.com/facebookresearch/fairseq/tree/main/examples/xmod).

Because it has been pre-trained with language-specific modular components (_language adapters_), X-MOD differs from previous multilingual models like [XLM-R](https://huggingface.co/xlm-roberta-base). For fine-tuning, the language adapters in each transformer layer are frozen.

# Usage

## Tokenizer
This model reuses the tokenizer of [XLM-R](https://huggingface.co/xlm-roberta-base), so you can load the tokenizer as follows:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
```

## Input Language
Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:

```python
from transformers import XMODModel

model = XMODModel.from_pretrained("jvamvas/xmod-base")
model.set_default_language("en_XX")
```

A directory of the language adapters in this model is found at the bottom of this model card.

## Fine-tuning
The paper recommends that the embedding layer and the language adapters are frozen during fine-tuning. A method for doing this is provided in the code:

```python
model.freeze_embeddings_and_language_adapters()
# Fine-tune the model ...
```

## Cross-lingual Transfer
After fine-tuning, zero-shot cross-lingual transfer can be tested by activating the language adapter of the target language:
```python
model.set_default_language("de_DE")
# Evaluate the model on German examples ...
```

# Bias, Risks, and Limitations

Please refer to the model card of [XLM-R](https://huggingface.co/xlm-roberta-base), because X-MOD has a similar architecture and has been trained on similar training data.


# Citation

**BibTeX:**

```bibtex
@inproceedings{pfeiffer-etal-2022-lifting,
    title = "Lifting the Curse of Multilinguality by Pre-training Modular Transformers",
    author = "Pfeiffer, Jonas  and
      Goyal, Naman  and
      Lin, Xi  and
      Li, Xian  and
      Cross, James  and
      Riedel, Sebastian  and
      Artetxe, Mikel",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.255",
    doi = "10.18653/v1/2022.naacl-main.255",
    pages = "3479--3495"
}
```

# Languages

This model contains the following language adapters:

| Language code | Language              |
|---------------|-----------------------|
| af_ZA         | Afrikaans             |
| am_ET         | Amharic               |
| ar_AR         | Arabic                |
| az_AZ         | Azerbaijani           |
| be_BY         | Belarusian            |
| bg_BG         | Bulgarian             |
| bn_IN         | Bengali               |
| ca_ES         | Catalan               |
| cs_CZ         | Czech                 |
| cy_GB         | Welsh                 |
| da_DK         | Danish                |
| de_DE         | German                |
| el_GR         | Greek                 |
| en_XX         | English               |
| eo_EO         | Esperanto             |
| es_XX         | Spanish               |
| et_EE         | Estonian              |
| eu_ES         | Basque                |
| fa_IR         | Persian               |
| fi_FI         | Finnish               |
| fr_XX         | French                |
| ga_IE         | Irish                 |
| gl_ES         | Galician              |
| gu_IN         | Gujarati              |
| ha_NG         | Hausa                 |
| he_IL         | Hebrew                |
| hi_IN         | Hindi                 |
| hr_HR         | Croatian              |
| hu_HU         | Hungarian             |
| hy_AM         | Armenian              |
| id_ID         | Indonesian            |
| is_IS         | Icelandic             |
| it_IT         | Italian               |
| ja_XX         | Japanese              |
| ka_GE         | Georgian              |
| kk_KZ         | Kazakh                |
| km_KH         | Central Khmer         |
| kn_IN         | Kannada               |
| ko_KR         | Korean                |
| ku_TR         | Kurdish               |
| ky_KG         | Kirghiz               |
| la_VA         | Latin                 |
| lo_LA         | Lao                   |
| lt_LT         | Lithuanian            |
| lv_LV         | Latvian               |
| mk_MK         | Macedonian            |
| ml_IN         | Malayalam             |
| mn_MN         | Mongolian             |
| mr_IN         | Marathi               |
| ms_MY         | Malay                 |
| my_MM         | Burmese               |
| ne_NP         | Nepali                |
| nl_XX         | Dutch                 |
| no_XX         | Norwegian             |
| or_IN         | Oriya                 |
| pa_IN         | Punjabi               |
| pl_PL         | Polish                |
| ps_AF         | Pashto                |
| pt_XX         | Portuguese            |
| ro_RO         | Romanian              |
| ru_RU         | Russian               |
| sa_IN         | Sanskrit              |
| si_LK         | Sinhala               |
| sk_SK         | Slovak                |
| sl_SI         | Slovenian             |
| so_SO         | Somali                |
| sq_AL         | Albanian              |
| sr_RS         | Serbian               |
| sv_SE         | Swedish               |
| sw_KE         | Swahili               |
| ta_IN         | Tamil                 |
| te_IN         | Telugu                |
| th_TH         | Thai                  |
| tl_XX         | Tagalog               |
| tr_TR         | Turkish               |
| uk_UA         | Ukrainian             |
| ur_PK         | Urdu                  |
| uz_UZ         | Uzbek                 |
| vi_VN         | Vietnamese            |
| zh_CN         | Chinese (simplified)  |
| zh_TW         | Chinese (traditional) |