README.md

---
license: cc-by-nc-4.0
tags:
- merge
---

Just 11B RP experemental merge. For some time I wanted to do something similar. 

## Model used
- [Sao10K/Fimbulvetr-11B-v2](https://huggingface.co/Sao10K/Fimbulvetr-11B-v2)
- [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2)
- [KatyTheCutie/LemonadeRP-4.5.3](https://huggingface.co/KatyTheCutie/LemonadeRP-4.5.3)
- [KoboldAI/Mistral-7B-Holodeck-1](https://huggingface.co/KoboldAI/Mistral-7B-Holodeck-1)
- [KoboldAI/Mistral-7B-Erebus-v3](https://huggingface.co/KoboldAI/Mistral-7B-Erebus-v3)

## Prompt template

Just use alpaca. Or you can try chatml, But I only tested this with alpaca and it works fine.

## The secret sauce

lemolake-11B :
```
slices:
  - sources:
    - model: KatyTheCutie/LemonadeRP-4.5.3
      layer_range: [0, 24]
  - sources:
    - model: senseable/WestLake-7B-v2
      layer_range: [8, 32]

merge_method: passthrough
dtype: bfloat16
```

holobus-11B :
```
slices:
  - sources:
    - model: KoboldAI/Mistral-7B-Holodeck-1
      layer_range: [0, 24]
  - sources:
    - model: KoboldAI/Mistral-7B-Erebus-v3
      layer_range: [8, 32]

merge_method: passthrough
dtype: bfloat16
```

Vertilake-11B :
```
base_model: "Mistral-11B-v0.1"
models:
  - model: "Mistral-11B-v0.1"
    # no parameters necessary for base model
  - model: "Fimbulvetr-11B-v2"
    parameters:
      weight: 0.43
      density: 0.8
  - model: "lemolake-11b"
    parameters:
      weight: 0.6
      density: 0.8
  - model: "Holobus-11B"
    parameters:
      weight: 0.17
      density: 0.5
merge_method: dare_ties
parameters:
  int8_mask: true
dtype: bfloat16

```
I use [mergekit](https://github.com/cg123/mergekit) for all the manipulation told here.

Thanks to the [Undi95](https://huggingface.co/Undi95) for the original [11B mistral merge](https://huggingface.co/Undi95/Mistral-11B-OmniMix) recipe.