Update README.md

README.md

@@ -1,3 +1,79 @@
 ---
-license: agpl-3.0
+license: apache-2.0
+library_name: peft
+tags:
+- not-for-all-audiences
 ---
+
+# LimaRP-ShareGPT-13b-loras
+
+This is a repository of my Llama-2-13b Qlora checkpoints based from the LimaRP dataset converted to ShareGPT.
+
+## Disclaimer
+
+This a **highly experimental** QLora test. If you want to use the LimaRP lora, please [look here instead](https://huggingface.co/lemonilia/limarp-llama2-v2). Lemonilia's Lora uses the Alpaca format.
+
+## Why?
+
+LimaRP is a high-quality lora with a dataset of human RP examples. However, it can come on too strong and nuke the personality of a character with a weight of 1.0. Therefore, lower weights are required when merging models. We wanted to see what would when formatting the dataset using shareGPT, a format that supports turn-based conversations instead of Alpaca which requires newline hackery.
+
+In addition, we wanted to see how various system prompts affect the end result of a lora finetune along with the use of character names as roles rather than the standard `USER` and `ASSISTANT`.
+
+## Roles
+- kingbri: Rewrote dataset creation script, trained all loras, reformatted dataset.
+- Alicat: Provided insight on system prompts and dataset formatting guidelines.
+- Argo: Provided insight on system prompts.
+
+## Variance
+
+For system prompts, please see the appropriate folder READMEs.
+
+One char = Character's persona in system prompt.
+Two char = Character and User's persona in system prompt.
+The scenario is always included.
+
+In addition, the dataprepare script randomizes a dataset before exporting it. This means that different portions were used for eval during training. Therefore each lora used a randomized version of LimaRP's 4k dataset. The randomization of what entries go to eval should not affect the results.
+
+## Notes
+
+These Qloras were produced as an experiment to see how varying versions of LimaRP can affect a model. Please take this data with a grain of salt. You are able to test these yourself and decide from there.
+
+### Architecture
+
+- **Model Architecture**: Llama-2-13b
+- **Training Algorithm**: QLora
+
+### Training Details
+
+- **Dataset**: LimaRP formatted with this [script](https://gist.github.com/bdashore3/4c9f3a812c1a68013fdb23e1179c7765)
+- **Datset type**: ShareGPT
+- **Training Parameters**: [See Here](https://gist.github.com/bdashore3/ab6cd21777a30fb9b131bc7b2f6b8949)
+- **Training Environment**: Axolotl
+- **sequence_len**: 4096
+
+## Instruct Format
+
+ShareGPT gets converted to vicuna format. The roles were character names when training, so there's no set role of `USER` or `ASSISTANT`. You can probably use the following and it should work:
+
+```
+SYSTEM: Enter roleplay mode...
+User: {prompt}
+Character:
+```
+
+Not using instruct mode (preferred) is also an option.
+
+## Acknowledgments
+
+Thanks to:
+- Lemonilia: Original creator of the LimaRP dataset and Lora
+- Axolotl: Finetuning suite
+
+## Donate?
+All my infrastructure and cloud expenses are paid out of pocket. If you'd like to donate, you can do so here: [https://ko-fi.com/kingbri](https://ko-fi.com/kingbri)
+
+You should not feel obligated to donate, but if you do, I'd appreciate it.
+
+## Axolotl stuff
+
+All axolotl stuff is located within each lora folder