README.md

metadata

license: cc-by-nc-4.0
tags:
  - not-for-all-audiences
  - nsfw

Description

This repo contains fp16 files of UtopiaXL-13B, a merge I have done with the new layer shuffle method from mergekit.

This is more a proof of concept showing the following:

• Llama2 is very flexible
• Llama2 don't care about what is finetuned on the layers specifically if you keep them in the same order
• Clean merge (no ties, no SLERP, etc...) with only layer is possible without breaking something
• Deleting special tokens/using model with special token don't break the model
• Alpaca win, always. So use it.

The name "XL" come from the absurd amount of model pushed into it.

Models and loras used

• Undi95/Utopia-13B
• KoboldAI/LLAMA2-13B-Holodeck-1
• Undi95/PsyMedRP-v1-13B
• PygmalionAI/pygmalion-2-13b
• Heralax/Cat-0.5
• KoboldAI/LLaMA2-13B-TiefighterLR
• Heralax/Augmental-13b-two-epochs
• Undi95/Storytelling-v2.1-13B-lora
• Undi95/LimaRP-UtopiaXL-13B-v3-lora

The sauce

!mergekit-layershuffle ./UtopiaXL \
  --model Undi95/Utopia-13B --weight 0.4 \
  --model KoboldAI/LLAMA2-13B-Holodeck-1 --weight 0.1 \
  --model Undi95/PsyMedRP-v1-13B --weight 0.1 \
  --model PygmalionAI/pygmalion-2-13b --weight 0.25 \
  --model Heralax/Cat-0.5 --weight 0.1 \
  --model KoboldAI/LLaMA2-13B-TiefighterLR --weight 0.1 \
  --model Heralax/Augmental-13b-two-epochs --weight 0.1 \
  --write-yaml UtopiaXL.yaml

=========================

merge_method: passthrough
slices:
- sources:
  - layer_range:
    - 0
    - 1
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 1
    - 2
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 2
    - 3
    model: KoboldAI/LLaMA2-13B-TiefighterLR
- sources:
  - layer_range:
    - 3
    - 4
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 4
    - 5
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 5
    - 7
    model: KoboldAI/LLaMA2-13B-TiefighterLR
- sources:
  - layer_range:
    - 7
    - 8
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 8
    - 11
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 11
    - 12
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 12
    - 13
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 13
    - 14
    model: KoboldAI/LLaMA2-13B-TiefighterLR
- sources:
  - layer_range:
    - 14
    - 15
    model: PygmalionAI/pygmalion-2-7b
- sources:
  - layer_range:
    - 15
    - 16
    model: KoboldAI/LLAMA2-13B-Holodeck-1
- sources:
  - layer_range:
    - 16
    - 17
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 17
    - 18
    model: Heralax/Cat-0.5
- sources:
  - layer_range:
    - 18
    - 19
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 19
    - 20
    model: KoboldAI/LLAMA2-13B-Holodeck-1
- sources:
  - layer_range:
    - 20
    - 22
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 22
    - 23
    model: KoboldAI/LLaMA2-13B-TiefighterLR
- sources:
  - layer_range:
    - 23
    - 25
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 25
    - 26
    model: KoboldAI/LLAMA2-13B-Holodeck-1
- sources:
  - layer_range:
    - 26
    - 27
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 27
    - 28
    model: PygmalionAI/pygmalion-2-7b
- sources:
  - layer_range:
    - 28
    - 29
    model: Heralax/Cat-0.5
- sources:
  - layer_range:
    - 29
    - 30
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 30
    - 32
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 32
    - 34
    model: PygmalionAI/pygmalion-2-7b
- sources:
  - layer_range:
    - 34
    - 36
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 36
    - 37
    model: KoboldAI/LLAMA2-13B-Holodeck-1
- sources:
  - layer_range:
    - 37
    - 38
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 38
    - 39
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 39
    - 40
    model: KoboldAI/LLAMA2-13B-Holodeck-1

=========================

=> Applying Undi95/Storytelling-v2.1-13B-lora x 0.1
=> Trained on LimaRP for +2h
=> Applying Undi95/LimaRP-UtopiaXL-13B-v3-lora x 0.35

Prompt template: Alpaca

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

If you want to support me, you can here.