File size: 2,700 Bytes
6cdab59 ed0b13e 6cdab59 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
---
license: apache-2.0
---
GGUF files for [Magic-Dolphin-7b](https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b)
---
# Magic-Dolphin-7b
<img src="https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b/resolve/main/magic-dolphin.jfif" width="500"/>
For fp16 files please look [here](https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b)
A linear merge of:
- [cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser)
- [Locutusque/Hyperion-1.5-Mistral-7B](https://huggingface.co/Locutusque/Hyperion-1.5-Mistral-7B)
- [ibm/merlinite-7b](https://huggingface.co/ibm/merlinite-7b)
These three models showed excellent acumen in technical topics so I wanted to see how they would behave together in a merge. Several different ratios were tested before this release, in the end a higher weighting for merlinite-7b helped smooth out some edges. This model is a test of how LAB tuning is impacted by merges with models leveraging DPO.
### Benchmark Performance
| Name | Avg. | ARC | HellaSwag | MMLU | TruthfulQA | Winograde | GSM8K |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| <b>Magic-Dolphin-7b</b> | <u><b>67.48</b></u> | 65.78 | 85.61 | 64.64 | 58.01 | <u><b>79.64</b></u> | <u><b>51.18</b></u> |
| dolphin-2.6-mistral-7b-dpo-laser | 67.28 | 66.3 | 85.73 | 63.16 | 61.71 | 79.16 | 47.61 |
| merlinite-7b | N/A | 63.99 | 84.37 | 64.88 | N/A | 78.24 | N/A |
| Hyperion-1.5-Mistral-7B | 61.43 | 60.49 | 83.64 | 63.57 | 41.78 | 78.61 | 40.49 |
This was my first experiment with merging models so any feedback is greatly appreciated.
Uses Alpaca template.
<p align="center">
</p>
<b>Sample Question</b>
<img src="https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b/resolve/main/magic-dolphin.JPG" width="750"/>
## Merge Details
### Merge Method
This model was merged using the [linear](https://arxiv.org/abs/2203.05482) merge method.
### Models Merged
The following models were included in the merge:
* [cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser)
* [Locutusque/Hyperion-1.5-Mistral-7B](https://huggingface.co/Locutusque/Hyperion-1.5-Mistral-7B)
* [ibm/merlinite-7b](https://huggingface.co/ibm/merlinite-7b)
### Configuration
The following YAML configuration was used to produce this model:
```yaml
models:
- model: models/dolphin-2.6-mistral-7b-dpo-laser
parameters:
weight: 1.0
- model: models/Hyperion-1.5-Mistral-7B
parameters:
weight: 0.3
- model: models/merlinite-7b
parameters:
weight: 0.5
merge_method: linear
dtype: float16
``` |