Create README.md

README.md

@@ -0,0 +1,283 @@
+---
+language: 
+- en
+- fr
+- ro
+- de
+- multilingual
+
+widget:
+- text: "Continue the dialogue as a task-oriented dialogue system called SYSTEM. The answer of SYSTEM should follow the ACTION provided next while answering the USER's last utterance: \n<USER> Hello, I am looking for a restaurant in Cambridge. I believe it is called Golden Wok. \n<EXTERNAL KNOWLEDGE> ACTION: {'Restaurant-Inform': [['address', '191 Histon Road Chesterton']]}"
+  example_title: "Dialog Act to Response Generation"
+- text: "Translate to German:  My name is Arthur"
+  example_title: "Translation"
+- text: "Please answer to the following question. Who is going to be the next Ballon d'or?"
+  example_title: "Question Answering"
+- text: "Q: Can Geoffrey Hinton have a conversation with George Washington? Give the rationale before answering."
+  example_title: "Logical reasoning"
+- text: "Please answer the following question. What is the boiling point of Nitrogen?"
+  example_title: "Scientific knowledge"
+- text: "Answer the following yes/no question. Can you write 200 words in a single tweet?"
+  example_title: "Yes/no question"
+- text: "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
+  example_title: "Reasoning task"
+- text: "Q: Is the statement ( `Jianguo is a research scientist at Salesforce AI` and `Jianguo is a student at UIC` ) True or Flase? A: Let's think step by step"
+  example_title: "Boolean Expressions"
+- text: "The square root of x is the cube root of y. What is y to the power of 2, if x = 4?"
+  example_title: "Math reasoning"
+- text: "Premise:  At my age you will probably have learnt one lesson. Hypothesis:  It's not certain how many lessons you'll learn by your thirties. Does the premise entail the hypothesis?"
+  example_title: "Premise and hypothesis"
+
+inference:
+  parameters:
+    max_length: 256
+
+tags:
+  - text2text-generation
+  - dialog
+
+datasets:
+- Salesforce/dialogstudio
+- flan
+
+
+license: apache-2.0
+---
+
+# Model Card for DialogStudio-T5 large
+
+<img src="https://huggingface.co/datasets/Salesforce/dialogstudio/resolve/main/logo.png"
+alt="drawing" width="510"/>
+
+#  Table of Contents
+
+0. [TL;DR](#TL;DR)
+1. [Model Details](#model-details)
+2. [Usage](#usage)
+3. [Uses](#uses)
+4. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+5. [Training Details](#training-details)
+6. [Evaluation](#evaluation)
+7. [Environmental Impact](#environmental-impact)
+8. [Citation](#citation)
+9. [Model Card Authors](#model-card-authors)
+
+# TL;DR
+
+If you already know T5 and Flan-T5, DialogStudio-T5 is better at many things. With the same number of parameters, the models are fine-tuned from a selected amount of dialogues from [DialogStudio](https://github.com/salesforce/DialogStudio) and also 1000 additional tasks. 
+
+
+**Disclaimer**: Content from **this** model card are modified from contents written by the Hugging Face team, and parts of it were copy pasted from the [T5 model card](https://huggingface.co/t5-large) and [Flan-T5 model card](https://huggingface.co/google/flan-t5-large).
+
+
+**Follow the [DialogStudio](https://github.com/salesforce/DialogStudio) GitHub repository for latest information.**
+
+
+# Model Details
+## Data
+
+We sample a small amount of dialogues from each commercial supported dataset under three categories of [DialogStudio](https://huggingface.co/datasets/Salesforce/dialogstudio), i.e., KG-Dial, TOD and Open-Domain dialogues. Additionally, we sample at most 150 examples for each non-translation task from [FLAN](https://github.com/google-research/FLAN/tree/main/flan/v2). 
+
+Note that this version does not incorporate datasets utilized for training large-scale models (>=7B) like Alpaca, ShareGPT, GPT4ALL, UltraChat from OpenAI's 'GPT-3.5/4', or other datasets such as OASST1 and WizardCoder.
+
+
+<img src="https://huggingface.co/datasets/Salesforce/dialogstudio/resolve/main/DialogStudio_Stats.jpg"
+alt="drawing" width="700"/>
+
+
+
+## Model Description
+
+
+- **Model type:** Language model
+- **Language(s) (NLP):** English, Spanish, Japanese, Persian, Hindi, French, Chinese, Bengali, Gujarati, German, Telugu, Italian, Arabic, Polish, Tamil, Marathi, Malayalam, Oriya, Panjabi, Portuguese, Urdu, Galician, Hebrew, Korean, Catalan, Thai, Dutch, Indonesian, Vietnamese, Bulgarian, Filipino, Central Khmer, Lao, Turkish, Russian, Croatian, Swedish, Yoruba, Kurdish, Burmese, Malay, Czech, Finnish, Somali, Tagalog, Swahili, Sinhala, Kannada, Zhuang, Igbo, Xhosa, Romanian, Haitian, Estonian, Slovak, Lithuanian, Greek, Nepali, Assamese, Norwegian
+- **License:** Apache 2.0
+- **Related Models:** [All DialogStudio-T5 Checkpoints](https://huggingface.co/models?search=dialogstudio-t5)
+- **Resources for more information:**
+  - [Research paper](https://arxiv.org/abs/2307.10172)
+  - [GitHub Repo](https://github.com/salesforce/DialogStudio)
+- **Maximum model length:**: 
+  - Maximum input length: 1200
+  - Maximum output length: 256
+- **Training formats:**
+  - We process dialogue data into below input format :
+    - With instruction and external knowledge: ```Instruction: your instruction <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N <EXTERNAL KNOWLEDGE> your external knowledge```
+    - Without instruction: ```<USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N <EXTERNAL KNOWLEDGE> your external knowledge```
+    - Without external knowledge: ```Instruction: your instruction <USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N```
+    - Without both: ```<USER> user utterance 1 <SYSTEM> system utterance 1 ... <USER> user utterance N```
+    - Note:  output is final the system response; `<USER>`, `<SYSTEM>` and `<EXTERNAL KNOWLEDGE>` are special tokens
+  - For sampled FLAN data:
+    - We follow their original data format, i.e., we did not set special tokens to separate in-context learning examples.
+  - In summary:
+    - We recommend you use our format and add our special tokens (such as `<USER>` and `<SYSTEM>` ) to get better performance. However, you may not necessary need to exactly follow our format if you do observe random behavios.
+    - We found that T5 model series such as Flan-t5 and DialogStudio-T5 may generate repetitive tokens during inference. If you find such repetition issues, you can set the `repetition_penalty` in model.generate(), such as 1.5, to mitigate them. Note that `repetition_penalty=1.0` by default.
+# Usage
+
+Find below some example scripts on how to use the model in `transformers`:
+
+## Using the Pytorch model
+
+### Running the model on a CPU
+
+<details>
+<summary> Click to expand </summary>
+
+```python
+
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0")
+model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0")
+
+input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids
+
+outputs = model.generate(input_ids, max_new_tokens=256)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+</details>
+
+### Running the model on a GPU
+
+<details>
+<summary> Click to expand </summary>
+
+```python
+# pip install accelerate
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0")
+model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0", device_map="auto")
+
+input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=256)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+</details>
+
+### Running the model on a GPU using different precisions
+
+#### FP16
+
+<details>
+<summary> Click to expand </summary>
+
+```python
+# pip install accelerate
+import torch
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0")
+model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0", device_map="auto", torch_dtype=torch.float16)
+
+input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=256)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+</details>
+
+#### INT8
+
+<details>
+<summary> Click to expand </summary>
+
+```python
+# pip install bitsandbytes accelerate
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+tokenizer = AutoTokenizer.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0")
+model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/dialogstudio-t5-large-v1.0", device_map="auto", load_in_8bit=True)
+
+input_text = "Answer the following yes/no question by reasoning step-by-step. Can you write 200 words in a single tweet?"
+input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=256)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+</details>
+
+# Uses
+
+## Direct Use and Downstream Use
+
+<!-- The authors write in [the original paper's model card](https://arxiv.org/pdf/2210.11416.pdf) that:  -->
+
+> The primary use is research on language models, including: research on zero-shot NLP tasks and in-context few-shot learning NLP tasks, such as dialogue response generation, reasoning, and question answering; advancing fairness and safety research, and understanding limitations of current large language models
+
+
+## Out-of-Scope Use
+
+More information needed.
+
+# Bias, Risks, and Limitations
+
+The information below in this section are copied and modified from Flan-T5's models card:
+
+> Language models, including DialogStudio-T5, can potentially be used for language generation in a harmful way, according to Rae et al. (2021). DialogStudio-T5 should not be used directly in any application, without a prior assessment of safety and fairness concerns specific to the application.
+
+## Ethical considerations and risks
+
+> DialogStudio-T5 is fine-tuned on a large corpus of text data that was not filtered for explicit content or assessed for existing biases. As a result the model itself is potentially vulnerable to generating equivalently inappropriate content or replicating inherent biases in the underlying data.
+
+## Known Limitations
+
+> DialogStudio-T5 has not been tested in real world applications.
+
+## Sensitive Use:
+
+> DialogStudio-T5 should not be applied for any unacceptable use cases, e.g., generation of abusive speech.
+
+# Training Details
+
+## Training Data
+
+We sample a small amount of dialogues from each commercial supported dataset under three categories of [DialogStudio](https://huggingface.co/datasets/Salesforce/dialogstudio), i.e., KG-Dial, TOD and Open-Domain dialogues. Additionally, we sample at most 150 examples for each non-translation task from [FLAN](https://github.com/google-research/FLAN/tree/main/flan/v2). 
+
+Note that this version does not incorporate datasets utilized for training large-scale models (>=7B) like Alpaca, ShareGPT, GPT4ALL, UltraChat from OpenAI's 'GPT-3.5/4', or other datasets such as OASST1 and WizardCoder.
+
+
+See above **Training formats:** for details of the training formats.
+
+## Training Procedure
+
+
+> These models are based on Flan-T5 and are fine-tuned with instructions for better zero-shot and few-shot performance. There is one fine-tuned DialogStudio model per T5 model size.
+
+The model has been trained on 16 A100 GPUs, each with 40G memory, using public [transformer](https://github.com/huggingface/transformers) codebase.
+
+
+# Evaluation
+
+## Testing Data, Factors & Metrics
+
+The authors evaluated the model on several dialogue tasks and general tasks such as 0-shot/5-shot MMLU and 3-shot BBH. 
+
+## Results 
+
+For full results for DialogStudio, see the [research paper](https://arxiv.org/abs/2307.10172).
+
+## Environmental Impact
+More information needed.
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@misc{zhang2023dialogstudio,
+      title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI}, 
+      author={Jianguo Zhang and Kun Qian and Zhiwei Liu and Shelby Heinecke and Rui Meng and Ye Liu and Zhou Yu and and Huan Wang and Silvio Savarese and Caiming Xiong},
+      year={2023},
+      eprint={2307.10172},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```