sultan commited on
Commit
c1924e2
1 Parent(s): 49171ae

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +36 -4
README.md CHANGED
@@ -1,14 +1,46 @@
1
  ArabicTransformer small model (B6-6-6 with decoder)
2
 
3
- <b>Paper</b> : ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective (EMNLP21)
4
 
5
- <b>Abstract</b>
 
 
6
 
7
  Pre-training Transformer-based models such as BERT and ELECTRA on a collection of Arabic corpora, demonstrated by both AraBERT and AraELECTRA, shows an impressive result on downstream tasks. However, pre-training Transformer-based language models is computationally expensive, especially for large-scale models. Recently, Funnel Transformer has addressed the sequential redundancy inside Transformer architecture by compressing the sequence of hidden states, leading to a significant reduction in the pretraining cost. This paper empirically studies the performance and efficiency of building an Arabic language model with Funnel Transformer and ELECTRA objective. We find that our model achieves state-of-the-art results on several Arabic downstream tasks despite using less computational resources compared to other BERT-based models.
8
 
9
- <b>Description</b>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
 
11
- This model was pre-trained on 44GB of Arabic corpora using [Funnel Transformer with ELECTRA objective](https://arxiv.org/abs/2006.03236). This model has more parameters (1.39x) than ELECTRA-base architecture while having similar or slightly larger inference and fine-tuning time. The model was pre-trained with significantly less resources than state-of-the-art models. We will update you with more details about the model and our accepted paper later at EMNLP21. Check our GitHub page for the latest updates and examples: https://github.com/salrowili/ArabicTransformer
12
 
13
  ```bibtex
14
  @inproceedings{alrowili-shanker-2021-arabictransformer-efficient,
 
1
  ArabicTransformer small model (B6-6-6 with decoder)
2
 
3
+ # Paper :
4
 
5
+ [ArabicTransformer: Efficient Large Arabic Language Model with Funnel Transformer and ELECTRA Objective](https://aclanthology.org/2021.findings-emnlp.108/)
6
+
7
+ # Abstract
8
 
9
  Pre-training Transformer-based models such as BERT and ELECTRA on a collection of Arabic corpora, demonstrated by both AraBERT and AraELECTRA, shows an impressive result on downstream tasks. However, pre-training Transformer-based language models is computationally expensive, especially for large-scale models. Recently, Funnel Transformer has addressed the sequential redundancy inside Transformer architecture by compressing the sequence of hidden states, leading to a significant reduction in the pretraining cost. This paper empirically studies the performance and efficiency of building an Arabic language model with Funnel Transformer and ELECTRA objective. We find that our model achieves state-of-the-art results on several Arabic downstream tasks despite using less computational resources compared to other BERT-based models.
10
 
11
+ # Description
12
+
13
+ This model was pre-trained on 44GB of Arabic corpora using [Funnel Transformer with ELECTRA objective](https://arxiv.org/abs/2006.03236). This model has more parameters (1.39x) than ELECTRA-base architecture while having similar or slightly larger inference and fine-tuning time. The model was pre-trained with significantly less resources than state-of-the-art models.
14
+
15
+ # Results on Arabic TyDi QA
16
+
17
+ | Model | EM | F1 |
18
+ |------------------|----|----|
19
+ | AraBERT02-Large | 73.72 | 86.03 |
20
+ | AraELECTRA-Base | 74.91 | 86.68 |
21
+ | ArabicTransformer-Small | 74.70 | 85.89 |
22
+ | ArabicTransformer-intermediate | 75.57 | 87.22 |
23
+
24
+
25
+ # Google Colab Examples
26
+
27
+ - Text Classification with ArabicTransformer with PyTorchXLA on TPU or with PyTorch on GPU (Better reproducibility but slower). [![Open In Colab][COLAB]](https://colab.research.google.com/github/salrowili/ArabicTransformer/blob/main/Examples/Text_Classification_with_ArabicTransformer_with_PyTorchXLA_on_TPU_or_with_PyTorch_on_GPU.ipynb)
28
+
29
+ - Text Classification with ArabicTransformer and TPU and Keras API (Faster but reproducibility is not better than PyTorchXLA).
30
+ [![Open In Colab][COLAB]](https://colab.research.google.com/github/salrowili/ArabicTransformer/blob/main/Examples/Text_Classification_with_ArabicTransformer_and_TPU_and_Keras_API.ipynb)
31
+
32
+ - Question Answering with ArabicTransformer.
33
+ [![Open In Colab][COLAB]](https://colab.research.google.com/github/salrowili/ArabicTransformer/blob/main/Examples/FineTuning_ArabicTransformers_on_TyDi_QA_and_Quran_QA_Dataset.ipynb)
34
+
35
+ [COLAB]: https://colab.research.google.com/assets/colab-badge.svg
36
+ # GitHub Page
37
+
38
+ https://github.com/salrowili/ArabicTransformer
39
+
40
+ # Acknowledgment
41
+
42
+ We would like to acknowledge the support we have from The TPU Research Cloud (TRC) team to grant us access to TPUv3 units.
43
 
 
44
 
45
  ```bibtex
46
  @inproceedings{alrowili-shanker-2021-arabictransformer-efficient,