BlueNipples
commited on
Commit
•
1f6aa2f
1
Parent(s):
57fb995
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,44 @@
|
|
1 |
---
|
2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: apache-2.0
|
3 |
+
tags:
|
4 |
+
- Roleplay
|
5 |
+
- Solar
|
6 |
+
- Mistral
|
7 |
+
- Text Generation
|
8 |
---
|
9 |
+
![SnowLotus Logo](https://cdn-uploads.huggingface.co/production/uploads/64bb1109aaccfd28b023bcec/gTQtPK46laLIFg0RTAv73.png)
|
10 |
+
|
11 |
+
### Premise
|
12 |
+
|
13 |
+
So this is a basic slerp merge between a smart model and a good prose model. Prose and smarts. What we all want in an uncensored RP model right? I feel like Solar has untapped potential, in any case.
|
14 |
+
|
15 |
+
Sao10K's Frostwind finetune is a key component of the mixture, its smarts are impressive. NyxKrage's Frostmaid experiment, which merges Frostwind with a frankenmerge of Noromaid and a mystery medical model, delivers quite impressive prose. His model creatively incorporates long-range context and instructions too, despite being slightly incoherent due to the fraken merging.
|
16 |
+
|
17 |
+
So those are the main ingredients. Thanks to Nyx for sorting out the pytorch files btw.
|
18 |
+
|
19 |
+
### Recipe
|
20 |
+
|
21 |
+
So, the recipe. I added solardoc by Nyx to frostwind at a 0.15 weight, and the gradient SLERP'd Frostwind (+solardoc) into Frostmaid with these params:
|
22 |
+
|
23 |
+
- filter: self_attn
|
24 |
+
value: [0.9, 0.4, 0.1, 0, 0]
|
25 |
+
- filter: mlp
|
26 |
+
value: [0.05, 0.95]
|
27 |
+
- value: 0.45
|
28 |
+
|
29 |
+
|
30 |
+
### Tentative Dozen or So Test Conclusion
|
31 |
+
|
32 |
+
This model seems to have better prose, less GPT-ish language and no degredation in coherency from the last version.
|
33 |
+
|
34 |
+
Cheers to all the finetuners, mergers and developers without which open source models wouldn't be half of what they are.
|
35 |
+
|
36 |
+
Resources used:
|
37 |
+
|
38 |
+
https://huggingface.co/NyxKrage/FrostMaid-10.7B-TESTING-pt
|
39 |
+
|
40 |
+
https://huggingface.co/Sao10K/Frostwind-10.7B-v1
|
41 |
+
|
42 |
+
https://huggingface.co/NyxKrage/Solar-Doc-10.7B-Lora
|
43 |
+
|
44 |
+
https://github.com/cg123/mergekit/tree/main
|