Create README.md

README.md

@@ -0,0 +1,36 @@
+---
+datasets:
+- SirNeural/flan_v2
+metrics:
+- perplexity
+tags:
+- flan
+- opt
+- peft
+---
+
+## FLAN-OPT-6.7b-LoRA
+
+OPT was first introduced in [Open Pre-trained Transformer Language Models](https://arxiv.org/abs/2205.01068) and first released in [metaseq's repository](https://github.com/facebookresearch/metaseq) on May 3rd 2022 by Meta AI.
+
+This model is [facebook/opt-6.7b](https://hf.co/facebook/opt-6.7b) finetuned with low-rank adapters (https://arxiv.org/abs/2106.09685) on the FLAN datasets (https://arxiv.org/pdf/2210.11416.pdf).
+
+Low-rank adapters (r=8) finetuned over 1.6m new tokens of a FLAN task mixture, with the start of each example cut off if it was too large to fit within a 256 token context.
+
+The model reaches a train ppl of 4.36 and an eval ppl of 4.32.
+
+### Example COT (Chain-of-thought) Prompt:
+
+```
+Q: Answer the following yes/no question by reasoning step-by-step. Could a dandelion suffer from hepatitis?
+A: Hepatitis only affects organisms with livers. Dandelions don’t have a liver. The answer is no.
+
+Q: Answer the following yes/no question by reasoning step-by-step. Can you write a whole Haiku in a single tweet?
+A: A haiku is a japanese three-line poem. That is short enough to fit in 280 characters. The answer is yes.
+
+Q: Answer the following yes/no question by reasoning step-by-step. Can you reach space with a Cessna?
+A:
+```
+``` 
+> A Cessna is a small plane. A small plane can't get into space. The answer is no.
+```