metadata
license: apache-2.0
tags:
- merge
- mergekit
- lazymergekit
- allenai/led-base-16384
- allenai/led-base-16384
LaMa-Merged-slerp
LaMa-Merged-slerp is a merge of the following models using mergekit:
🧩 Configuration
slices:
- sources:
- model: allenai/led-base-16384
layer_range: [0, 12] # Taking the initial layers from LED Base model
- model: allenai/led-base-16384
layer_range: [12, 24] # Taking the later layers from LED Base model
merge_method: slerp
base_model: allenai/led-base-16384 # Using LED Base model as the base model
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1] # Interpolation values for self-attention layers
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0] # Interpolation values for MLP layers
- value: 0.5 # Default interpolation value
dtype: bfloat16