Transformer QG on SQuAD
The inputs of the model refers to
we integrate C and A into a new C' in the following form.
C' = [c1, c2, ..., [HL], a1, ..., a|A|, [HL], ..., c|C|]
Proposed by Ying-Hong Chan & Yao-Chung Fan. (2019). A Re-current BERT-based Model for Question Generation.
More detail: p208p2002/Transformer-QG-on-SQuAD
Features
- Fully pipline from fine-tune to evaluation
- Support most of state of the art models
- Fast deploy as a API server
Data setting
We report two dataset setting as Follow
SQuAD
- train: 87599
- validation: 10570
SQuAD NQG
- train: 75722
- dev: 10570
- test: 11877
Learning to Ask: Neural Question Generation for Reading Comprehension
Available models
- BART
- GPT2
- T5
Expriments
We report score with NQG Scorer which is using in SQuAD NQG.
If not special explanation, the size of the model defaults to "base".
SQuAD
| Model | Bleu 1 | Bleu 2 | Bleu 3 | Bleu 4 | METEOR | ROUGE-L |
|---|---|---|---|---|---|---|
| BART-HLSQG | 54.67 | 39.26 | 30.34 | 24.15 | 25.43 | 52.64 |
| GPT2-HLSQG | 49.31 | 33.95 | 25.41 | 19.69 | 22.29 | 48.82 |
| T5-HLSQG | 54.29 | 39.22 | 30.43 | 24.26 | 25.56 | 53.11 |
SQuAD NQG
| Model | Bleu 1 | Bleu 2 | Bleu 3 | Bleu 4 | METEOR | ROUGE-L |
|---|---|---|---|---|---|---|
| BERT-HLSQG (Chan et al.) | 49.73 | 34.60 | 26.13 | 20.33 | 23.88 | 48.23 |
| BART-HLSQG | 54.12 | 38.19 | 28.84 | 22.35 | 24.55 | 51.03 |
| GPT2-HLSQG | 49.82 | 33.69 | 24.71 | 18.63 | 21.90 | 47.60 |
| T5-HLSQG | 53.13 | 37.60 | 28.62 | 22.38 | 24.48 | 51.20 |