---
license: apache-2.0
library_name: peft
tags:
- not-for-all-audiences
---

# LimaRP-ShareGPT-13b-loras

This is a repository of my Llama-2-13b Qlora checkpoints based from the LimaRP dataset converted to ShareGPT.

## Disclaimer

This a **highly experimental** QLora test. If you want to use the LimaRP lora, please [look here instead](https://huggingface.co/lemonilia/limarp-llama2-v2). Lemonilia's Lora uses the Alpaca format.

## Why?

LimaRP is a high-quality lora with a dataset of human RP examples. However, it can come on too strong and nuke the personality of a character with a weight of 1.0. Therefore, lower weights are required when merging models. We wanted to see what would when formatting the dataset using shareGPT, a format that supports turn-based conversations instead of Alpaca which requires newline hackery.

In addition, we wanted to see how various system prompts affect the end result of a lora finetune along with the use of character names as roles rather than the standard `USER` and `ASSISTANT`.

## Roles
- kingbri: Rewrote dataset creation script, trained all loras, reformatted dataset.
- Alicat: Provided insight on system prompts and dataset formatting guidelines.
- Argo: Provided insight on system prompts.

## Variance

For system prompts, please see the appropriate folder READMEs.

One char = Character's persona in system prompt.
Two char = Character and User's persona in system prompt.
The scenario is always included.

In addition, the dataprepare script randomizes a dataset before exporting it. This means that different portions were used for eval during training. Therefore each lora used a randomized version of LimaRP's 4k dataset. The randomization of what entries go to eval should not affect the results.

## Notes

These Qloras were produced as an experiment to see how varying versions of LimaRP can affect a model. Please take this data with a grain of salt. You are able to test these yourself and decide from there.

### Architecture

- **Model Architecture**: Llama-2-13b
- **Training Algorithm**: QLora

### Training Details

- **Dataset**: LimaRP formatted with this [script](https://gist.github.com/bdashore3/4c9f3a812c1a68013fdb23e1179c7765)
- **Datset type**: ShareGPT
- **Training Parameters**: [See Here](https://gist.github.com/bdashore3/ab6cd21777a30fb9b131bc7b2f6b8949)
- **Training Environment**: Axolotl
- **sequence_len**: 4096

## Instruct Format

ShareGPT gets converted to vicuna format. The roles were character names when training, so there's no set role of `USER` or `ASSISTANT`. You can probably use the following and it should work:

```
SYSTEM: Enter roleplay mode...
User: {prompt}
Character:
```

Not using instruct mode (preferred) is also an option.

## Acknowledgments

Thanks to:
- Lemonilia: Original creator of the LimaRP dataset and Lora
- Axolotl: Finetuning suite

## Donate?
All my infrastructure and cloud expenses are paid out of pocket. If you'd like to donate, you can do so here: [https://ko-fi.com/kingbri](https://ko-fi.com/kingbri)

You should not feel obligated to donate, but if you do, I'd appreciate it.

## Axolotl stuff

All axolotl stuff is located within each lora folder