Update README.md

README.md

@@ -4,33 +4,26 @@ language:
 tags:
 - causal-lm
 license: cc-by-nc-4.0
-datasets:
-- dmayhem93/ChatCombined
-- tatsu-lab/alpaca
-- nomic-ai/gpt4all_prompt_generations
-- Dahoas/full-hh-rlhf
-- jeffwan/sharegpt_vicuna
-- HuggingFaceH4/databricks_dolly_15k
 ---
 
-# StableLM-Tuned-Alpha
+# StableLM-Base-Alpha
 
 ## Model Description
 
-`StableLM-Tuned-Alpha` is a suite of 3B and 7B parameter decoder-only language models built on top of the `StableLM-Base-Alpha` models and further fine-tuned on various chat and instruction-following datasets.
+`StableLM-Base-Alpha` is a suite of 3B and 7B parameter decoder-only language models pre-trained on a diverse collection of English datasets with a sequence length of 4096 to push beyond the context window limitations of existing open-source language models.
 
 ## Usage
 
-Get started chatting with `StableLM-Tuned-Alpha` by using the following code snippet:
+Get started generating text with `StableLM-Base-Alpha` by using the following code snippet:
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
-model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-tuned-alpha-7b")
-model.half().cuda()
+tokenizer = AutoTokenizer.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
+model = AutoModelForCausalLM.from_pretrained("StabilityAI/stablelm-base-alpha-7b")
+model.half()
 
-inputs = tokenizer("What's your mood today?", return_tensors="pt").to('cuda')
+inputs = tokenizer("What's your mood today?", return_tensors="pt")
 tokens = model.generate(
   **inputs,
   max_new_tokens=64,
@@ -43,11 +36,11 @@ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
 ## Model Details
 
 * **Developed by**: [Stability AI](https://stability.ai/)
-* **Model type**: StableLM-Tuned-Alpha models are auto-regressive language models based on the NeoX transformer architecture.
+* **Model type**: StableLM-Base-Alpha models are auto-regressive language models based on the NeoX transformer architecture.
 * **Language(s)**: English
-* **Library**: [HuggingFace Transformers](https://github.com/huggingface/transformers)
-* **License**: [CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
-* **Contact**: For questions and comments about the model, please email {TODO: email address}
+* **Library**: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
+* **License**: [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)
+* **Contact**: For questions and comments about the model, please email **{TODO: FILL IN CONTACT ADDRESS}**
 
 ## Training
 
@@ -58,66 +51,32 @@ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
 
 ### Training Dataset
 
-`StableLM-Tuned-Alpha` models are fine-tuned on a combination of five datasets:
-[Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca), a dataset of 52,000 instructions and demonstrations generated by OpenAI's `text-davinci-003` engine.
-[GPT4All Prompt Generations](https://huggingface.co/datasets/nomic-ai/gpt4all_prompt_generations), which consists of 400k prompts and responses generated by GPT-4;
-[Anthropic HH](https://huggingface.co/datasets/Dahoas/full-hh-rlhf), made up of preferences about AI assistant helpfulness and harmlessness;
-[DataBricks Dolly](https://github.com/databrickslabs/dolly), comprising 15k instruction/responses generated by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization;
-and [ShareGPT Vicuna (English subset)](https://huggingface.co/datasets/jeffwan/sharegpt_vicuna), a dataset of conversations retrieved from [ShareGPT](https://sharegpt.com/).
+`StableLM-Base-Alpha` is pre-trained on a new experimental dataset built atop [The Pile](https://huggingface.co/datasets/EleutherAI/the_pile) and is threes times larger at approximately 1.5T tokens.
 
 ### Training Procedure
 
-Models are learned via supervised fine-tuning on the aforementioned datasets, trained in mixed-precision (FP16), and optimized with AdamW. We outline the following hyperparameters:
-
-| Parameters | Batch Size | Learning Rate | Warm-up | Weight Decay | Betas       |
-|------------|------------|---------------|---------|--------------|-------------|
-| 3B         | 256        | 2e-5          | 50      | 0.01         | (0.9, 0.99) |
-| 7B         | 128        | 2e-5          | 100     | 0.01         | (0.9, 0.99) |
+Models are pre-trained on the aforementioned dataset in mixed-precision (FP16), optimized with Adam, and trained using the NeoX tokenizer with a vocabulary size of 50,257. We outline the complete hyperparameters choices in the project's GitHub repository **{TODO: FILL IN LINK}**.
 
 ## Use and Limitations
 
 ### Intended Use
 
-These models are intended to be used by the open-source community chat-like applications in adherence with the [CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license.
+These models are intended to be used by all individuals as foundational models for application-specific fine-tuning without strict limitations on commercial use.
 
 ### Limitations and bias
 
-Although the aforementioned datasets help to steer the base language models into "safer" distributions of text, not all biases and toxicity can be mitigated through fine-tuning. We ask that users be mindful of such potential issues that can arise in generated responses. Do not treat model outputs as substitutes for human judgment or as sources of truth. Please use responsibly.
-
-## Acknowledgements
-
-This work would not have been possible without the helpful hand of Dakota Mahan ([@dmayhem93](https://huggingface.co/dmayhem93)).
+The pre-training dataset may contain offensive or inappropriate content even after applying data cleansing filters which can be reflected in generated text. We recommend users exercise reasonable caution when using these models in production systems. Do not use the models for any applications that may cause harm or distress to individuals or groups.
 
 ## Citations
 
-```bibtex
-@misc{alpaca,
-  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
-  title = {Stanford Alpaca: An Instruction-following LLaMA model},
-  year = {2023},
-  publisher = {GitHub},
-  journal = {GitHub repository},
-  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
-}
-```
-
 ```bibtext
-@misc{vicuna2023,
-    title = {Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality},
-    url = {https://vicuna.lmsys.org},
-    author = {Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P.},
-    month = {March},
-    year = {2023}
-}
-```
-
-```bibtex
-@misc{gpt4all,
-  author = {Yuvanesh Anand and Zach Nussbaum and Brandon Duderstadt and Benjamin Schmidt and Andriy Mulyar},
-  title = {GPT4All: Training an Assistant-style Chatbot with Large Scale Data Distillation from GPT-3.5-Turbo},
-  year = {2023},
-  publisher = {GitHub},
-  journal = {GitHub repository},
-  howpublished = {\url{https://github.com/nomic-ai/gpt4all}},
+@software{gpt-neox-library,
+  title = {{GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch}},
+  author = {Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel},
+  url = {https://www.github.com/eleutherai/gpt-neox},
+  doi = {10.5281/zenodo.5879544},
+  month = {8},
+  year = {2021},
+  version = {0.0.1},
 }
 ```