README.md

---
license: apache-2.0
tags:
- freeze
- generated_from_trainer
- autoquant
- exl2
base_model: mistralai/Mistral-7B-Instruct-v0.2
model-index:
- name: rubra-11b-h
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# rubra-11b-h

![image/png](./rubra-11b-h.png)

## Model description
This is a mistral based model trained on a lot of code and conversational data. The goal of this model was to retain the abilities of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) while making it better at reasoning.

* 32k context window
* Rope-theta = 1e6

Fine tuning instruct models has proven to be a challenge when the goal is to create a model that is not meant for a particular task. Specifically, fine tuning an instruct model often leads to destructive forgetting in the resulting model. This is why most fine tunes select a base (non-instruct) language model. The rubra-v0.1 series models aim to add desired traits like multi-turn chat capabilites while retaining knowledge found in instruct models

## Chat format

Your prompt should be surrounded by `[INST]` and `[/INST]` tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.

E.g.
```
text = "<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
"[INST] Do you have mayonnaise recipes? [/INST]"
```

This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("sanjay920/rubra-11b-h")
tokenizer = AutoTokenizer.from_pretrained("sanjay920/rubra-11b-h")

messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
```

### GGUF

GGUF quants are available at [sanjay920/rubra-11b-h-GGUF](https://huggingface.co/sanjay920/rubra-11b-h-GGUF)

Create a new discussion for other quant variant requests!

#### LM Studio
 Set "Chat Preset" to `Mistral Instruct`

## Training procedure
* Block expansion and further training

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 12
- total_train_batch_size: 12
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 3.0
- mixed_precision_training: Native AMP

### Training results
```
{
    "epoch": 3.0,
    "train_loss": 0.5844329195557438,
    "train_runtime": 307371.8718,
    "train_samples_per_second": 0.246,
    "train_steps_per_second": 0.021
}
```


### Framework versions

- Transformers 4.38.2
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2

## Evaluation Results

```
########## Average ##########
                             score
model                             
gpt-4-0613                9.020312  
gpt-3.5-turbo-0125        7.970219
rubra-11b-h               7.800000
Mistral-7B-Instruct-v0.2  7.793750
Mistral-7B-Instruct-v0.1  6.721875
Llama-2-13b-chat          6.650000
```

![image/png](./mtbench-comparison.png)

## Intended uses & limitations

This model is still under development. It hasn't been aligned for safety, so it may produce biased generations. It appears to produce thoughtful answers compared to other small LLMs. See below:

## Examples

![image/png](./example1.png)
![image/png](./example2.png)
![image/png](./example3.png)

### Needle in a haystack test:

The input text can be found in [needle-in-a-haystack.txt](./needle-in-a-haystack.txt)

* 14,334 input tokens
* "Random number is 23." inserted into the middle of the long document text. Asked the LLM what the random number is.

**rubra-11b-h**:
```
The random number is 23.
```
* Correct

**mistral-7b-instruct-v0.2**:
```
The random number generated from this text is 23. However, it's important to note that the method used to generate this number from the text is not a secure or reliable one and should not be used for any purpose where security is a concern. This was just a fun exercise to extract a number from the text.

As for the content of the text, it discusses the idea of creating a technology hub or "silicon valley" by attracting the right people - specifically, rich people and nerds. The author argues that these two groups are the limiting reagents in the reaction that produces startups, as they are the only ones present when startups get started. The text also touches on the idea that universities can act as magnets for these people and that personality and youth are important factors in creating a successful technology hub.
```
* Correct

**hermes-2-pro**:
```
 the<dummy00028>'1 Question1 question2<dummy00028> <dummy00015>..... in3<dummy00028>d<dummy00028>}<dummy00015>l kennis4<dummy00012>il<dummy00028>y<dummy00028>}2<dummy00028>-^{ a<dummy00028>}3<dummy00012>. a<dummy00028>}<dummy00015> toion<dummy00028>} people a a<dummy00028>}5<dummy00005>- a Question1<dummy00028>}y<dummy00028>}6<dummy00012>. a<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00012>,<dummy00015>l<dummy00028>},7<dummy00012>. <dummy00015>l Fuß in the<dummy00028>
 a a<dummy00028>'2 a<dummy00028>}4 ál a Question3<dummy00012>o kennis in if to more a<dummy00028> or a<dummy00028>'5<dummy00005> 1<dummy00028>}1<dummy00005>s<dummy00028>y<dummy00028>y<dummy00028>'6 Question9 Question1<dummy00028>'2 Questions<dummy00028>y biologie1<dummy00028>y<dummy00028>y<dummy00028>-3<dummy00028>'1 Question. a<dummy00028>* the<dummy00028>}<dummy00015>y Question a<dummy00028>4<dummy00012>,ll Question1 Question6<dummy00028>4<dummy00028>}2<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00028>
 if a Questionia kennisa<dummy00005> <dummy00015> a<dummy00028>h<dummy00028>}2<dummy00028>a égal
```
* Incorrect. Unusable output