Official repository: seonghyeonye/Flipped-Learning
Model Description
FLIPPED uses a unique meta-learning method to show zero-shot task generalization on classification natural language prompts, outperforming GPT-3 and T0-11B on many tasks with a 4x smaller scale. It is a series of encoder-decoder model trained on a numerous classification dataset. We show inputs and its corresponding outputs of each instances in each dataset to FLIPPED, and train it to generate its possible instruction. We add unlikelyhood loss in order not to generate the instruction when given the same input, but a wrong output. To obtain FLIPPED, we fine-tune a T5 model in a given scale on a multitask mixture covering many different classification NLP tasks.
Intended uses
You can use the models to perform inference on tasks by specifying your input-output NLP query in a "input: {input}\noutput: {output}" form , and the model will predict the instruction. For example, You can try "input: this is the best cast iron skillet you will ever buy\noutput: Positive" as an input, and the model will hopefully generate "Title: Review:".
How to use
Our overall explanation models along with ablations can be found in our paper. We recommend using the FLIPPED-11B checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.
| Model | Number of parameters |
|---|---|
| Flipped_11B | 11 billion |
| Flipped_3B | 3 billion |
| Here is how to use the model in PyTorch: |
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("seonghyeonye/flipped_3B")
model = AutoModelForSeq2SeqLM.from_pretrained("seonghyeonye/flipped_3B")
inputs = tokenizer.encode("input: this is the best cast iron skillet you will ever buy\noutput: Positive", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
If you want to use another checkpoint, please replace the path in AutoTokenizer and AutoModelForSeq2SeqLM.
Note: the model was trained with fp32 activations. As such, we highly discourage running inference with fp16.
Training procedure
FLIPPED models are based on T5, a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on C4. At a high level, the input text along with output label is fed to the encoder and the instruction text is produced by the decoder. The model is fine-tuned to autoregressively generate the target. We also feed input text along with a wrong input, adding an unlikelyhood loss in order not to make model produce the proper instruction in that case. Here are our training details. Training details:
- Fine-tuning steps: 5'000
- Input sequence length: 384(512 for 3B)
- Target sequence length: 64
- Batch size: 1
- Optimizer: Adafactor
- Learning rate: 5e-5
- Dropout: 0.1
- Sampling strategy: proportional to the number of examples in each dataset (we randomly sampled any dataset if it has over 500'000 examples so that it has at most 500'000 examples. Also, we randomly choose which instruction to generate for each training steps, so ideally each instruction appears num_examples/num_templates while training.)
Training data
We trained different variants T0 with different mixtures of datasets.
| Model | Training datasets |
|---|---|
| FLIPPED | - Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ - Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp - Topic Classification: AG News, DBPedia - Paraphrase Identification: MRPC, PAWS, QQP |
| FLIPPED_3B | Same as T0 but starting from a T5-LM (3B parameters) pre-trained model |
| We only choose prompts examples that has output lables, which can be found on the dataset page. |
Evaluation data
We evaluate our models on following datasets:
| Task category | Datasets |
|---|---|
| Natural language inference | ANLI(R1, R2, R3), CB, RTE |
| Coreference resolution | WSC, Winogrande |
| Word sense disambiguation | WiC |
| Sentence completion | COPA, HellaSwag, Story Cloze |
| QA | PIQA, ARC-Challenge, OpenbookQA |
| We also evaluate FLIPPED on a subset of BIG-bench benchmark: |
- Code description task
- Conceptual combinations
- Hindu knowledge json
- Known unknowns
- Language identification
- Logic grid puzzle task
- Logical deduction
- Common misconceptions
- Movie dialog same or different
- Novel concepts
- Strategyqa
- Formal fallacies syllogisms negation
- VitaminC
- Winowhy multiple choice
Label generalization
We evaluate the robustness of models on following datasets with changing the output label of the datasets. The substitute words can be found in our paper.
| Task category | (Datasets, Template name) |
|---|---|
| Unseen tasks | (WSC, does the pronoun refer to), (CB, can we infer), (RTE, MNLI crowdsource) |
| Seen tasks | (IMDB, Reviewer Enjoyment Yes No), (PAWS, Meaning) |
| The template name we used can be found in the promptsource template library. |
BibTeX entry and citation info
@misc{https://doi.org/10.48550/arxiv.2210.02969,
doi = {10.48550/ARXIV.2210.02969},
url = {https://arxiv.org/abs/2210.02969},
author = {Ye, Seonghyeon and Kim, Doyoung and Jang, Joel and Shin, Joongbo and Seo, Minjoon},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}