|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- squad_v2 |
|
- drop |
|
- mou3az/IT_QA-QG |
|
language: |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text2text-generation |
|
tags: |
|
- 'IT purpose ' |
|
- General purpose |
|
--- |
|
### Model Card ### |
|
|
|
# Information: |
|
|
|
Base Model: facebook/bart-base |
|
|
|
Fine-tuned : using PEFT-LoRa |
|
|
|
Datasets : squad_v2, drop, mou3az/IT_QA-QG |
|
|
|
Task: Generating questions from context and answers |
|
|
|
Language: English |
|
|
|
|
|
# Performance Metrics on Evaluation Set: |
|
|
|
Training Loss: 1.1.1958 |
|
|
|
Evaluation Loss: 1.109059 |
|
|
|
|
|
### Loading the model ### |
|
|
|
```python |
|
from peft import PeftModel, PeftConfig |
|
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer |
|
HUGGING_FACE_USER_NAME = "mou3az" |
|
model_name = "ITandGeneral_Question-Generation" |
|
peft_model_id = f"{HUGGING_FACE_USER_NAME}/{model_name}" |
|
config = PeftConfig.from_pretrained(peft_model_id) |
|
model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=False, device_map='auto') |
|
QG_tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path) |
|
QG_model = PeftModel.from_pretrained(model, peft_model_id) |
|
``` |
|
|
|
### At inference time ### |
|
|
|
```python |
|
def get_question(context, answer): |
|
device = next(L_model.parameters()).device |
|
input_text = f"Given the context '{context}' and the answer '{answer}', what question can be asked?" |
|
encoding = G_tokenizer.encode_plus(input_text, padding=True, return_tensors="pt").to(device) |
|
|
|
output_tokens = L_model.generate(**encoding, early_stopping=True, num_beams=5, num_return_sequences=1, no_repeat_ngram_size=2, max_length=100) |
|
out = G_tokenizer.decode(output_tokens[0], skip_special_tokens=True).replace("question:", "").strip() |
|
|
|
return out |
|
``` |
|
|
|
### Training parameters and hyperparameters ### |
|
|
|
The following were used during training: |
|
|
|
# For Lora: |
|
|
|
r=18 |
|
|
|
alpha=8 |
|
|
|
|
|
# For training arguments: |
|
|
|
gradient_accumulation_steps=24 |
|
|
|
per_device_train_batch_size=8 |
|
|
|
per_device_eval_batch_size=8 |
|
|
|
max_steps=1000 |
|
|
|
warmup_steps=50 |
|
|
|
weight_decay=0.05 |
|
|
|
learning_rate=3e-3 |
|
|
|
lr_scheduler_type="linear" |
|
|
|
### Training Results ### |
|
|
|
| Epoch | Training Loss | Validation Loss | |
|
|-------|---------------|-----------------| |
|
| 0.0 | 4.6426 | 4.704238 | |
|
| 3.0 | 1.5094 | 1.202135 | |
|
| 6.0 | 1.2677 | 1.146177 | |
|
| 9.0 | 1.2613 | 1.112074 | |
|
| 12.0 | 1.1958 | 1.109059 | |
