Hugging Face's logo

Fischerboot
/
llama3-sophie-adapter-model

PEFT
llama
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files Community
llama3-sophie-adapter-model / README.md
Fischerboot's picture
Fischerboot
uploaded ze files
368c15b verified
preview code
|
raw
history blame contribute delete
13.8 kB
---
base_model: Fischerboot/LLama3-Lexi-Aura-3Some-SLERP-SLERP-ql-merge
library_name: peft
tags:
- generated_from_trainer
model-index:
- name: outputs/qlora-out
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
<details><summary>See axolotl config</summary>
axolotl version: `0.4.1`
```yaml
base_model: Fischerboot/LLama3-Lexi-Aura-3Some-SLERP-SLERP-ql-merge
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
strict: false
chat_template: llama3
datasets:
- path: Fischerboot/dataset
type: sharegpt
conversation: llama3
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/qlora-out
adapter: qlora
lora_model_dir:
sequence_len: 128
sample_packing: false
pad_to_sequence_len: true
lora_r: 1024
lora_alpha: 512
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 50
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
eval_sample_packing: false
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: "<|begin_of_text|>"
eos_token: "<|end_of_text|>"
pad_token: "<|end_of_text|>"
```
</details><br>
# outputs/qlora-out
This model is a fine-tuned version of [Fischerboot/LLama3-Lexi-Aura-3Some-SLERP-SLERP-ql-merge](https://huggingface.co/Fischerboot/LLama3-Lexi-Aura-3Some-SLERP-SLERP-ql-merge) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0000
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 50
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:------:|:-----:|:---------------:|
| 5.9056 | 0.0030 | 1 | 5.7722 |
| 0.0002 | 0.25 | 82 | 0.0002 |
| 0.0262 | 0.5 | 164 | 0.0310 |
| 0.0004 | 0.75 | 246 | 0.1491 |
| 0.2739 | 1.0 | 328 | 0.0013 |
| 0.0005 | 1.25 | 410 | 0.0490 |
| 0.0 | 1.5 | 492 | 0.0000 |
| 0.0 | 1.75 | 574 | 0.0000 |
| 0.0 | 2.0 | 656 | 0.0000 |
| 0.0 | 2.25 | 738 | 0.0000 |
| 0.0 | 2.5 | 820 | 0.0000 |
| 0.0 | 2.75 | 902 | 0.0000 |
| 0.0 | 3.0 | 984 | 0.0000 |
| 0.0 | 3.25 | 1066 | 0.0000 |
| 0.0 | 3.5 | 1148 | 0.0000 |
| 0.0 | 3.75 | 1230 | 0.0000 |
| 0.0 | 4.0 | 1312 | 0.0000 |
| 0.0 | 4.25 | 1394 | 0.0000 |
| 0.0 | 4.5 | 1476 | 0.0000 |
| 0.0 | 4.75 | 1558 | 0.0000 |
| 0.0 | 5.0 | 1640 | 0.0000 |
| 0.0 | 5.25 | 1722 | 0.0000 |
| 0.0 | 5.5 | 1804 | 0.0000 |
| 0.0 | 5.75 | 1886 | 0.0000 |
| 0.0 | 6.0 | 1968 | 0.0000 |
| 0.0 | 6.25 | 2050 | 0.0000 |
| 0.0 | 6.5 | 2132 | 0.0000 |
| 0.0 | 6.75 | 2214 | 0.0000 |
| 0.0 | 7.0 | 2296 | 0.0000 |
| 0.0 | 7.25 | 2378 | 0.0000 |
| 0.0 | 7.5 | 2460 | 0.0000 |
| 0.0 | 7.75 | 2542 | 0.0000 |
| 0.0 | 8.0 | 2624 | 0.0000 |
| 0.0 | 8.25 | 2706 | 0.0000 |
| 0.0 | 8.5 | 2788 | 0.0000 |
| 0.0 | 8.75 | 2870 | 0.0000 |
| 0.0 | 9.0 | 2952 | 0.0000 |
| 0.0 | 9.25 | 3034 | 0.0000 |
| 0.0 | 9.5 | 3116 | 0.0000 |
| 0.0 | 9.75 | 3198 | 0.0000 |
| 0.0 | 10.0 | 3280 | 0.0000 |
| 0.0 | 10.25 | 3362 | 0.0000 |
| 0.0 | 10.5 | 3444 | 0.0000 |
| 0.0 | 10.75 | 3526 | 0.0000 |
| 0.0 | 11.0 | 3608 | 0.0000 |
| 0.0 | 11.25 | 3690 | 0.0000 |
| 0.0 | 11.5 | 3772 | 0.0000 |
| 0.0 | 11.75 | 3854 | 0.0000 |
| 0.0 | 12.0 | 3936 | 0.0000 |
| 0.0 | 12.25 | 4018 | 0.0000 |
| 0.0 | 12.5 | 4100 | 0.0000 |
| 0.0 | 12.75 | 4182 | 0.0000 |
| 0.0 | 13.0 | 4264 | 0.0000 |
| 0.0 | 13.25 | 4346 | 0.0000 |
| 0.0 | 13.5 | 4428 | 0.0000 |
| 0.0 | 13.75 | 4510 | 0.0000 |
| 0.0 | 14.0 | 4592 | 0.0000 |
| 0.0 | 14.25 | 4674 | 0.0000 |
| 0.0 | 14.5 | 4756 | 0.0000 |
| 0.0 | 14.75 | 4838 | 0.0000 |
| 0.0 | 15.0 | 4920 | 0.0000 |
| 0.0 | 15.25 | 5002 | 0.0000 |
| 0.0 | 15.5 | 5084 | 0.0000 |
| 0.0 | 15.75 | 5166 | 0.0000 |
| 0.0 | 16.0 | 5248 | 0.0000 |
| 0.0 | 16.25 | 5330 | 0.0000 |
| 0.0 | 16.5 | 5412 | 0.0000 |
| 0.0 | 16.75 | 5494 | 0.0000 |
| 0.0 | 17.0 | 5576 | 0.0000 |
| 0.0 | 17.25 | 5658 | 0.0000 |
| 0.0 | 17.5 | 5740 | 0.0000 |
| 0.0 | 17.75 | 5822 | 0.0000 |
| 0.0 | 18.0 | 5904 | 0.0000 |
| 0.0 | 18.25 | 5986 | 0.0000 |
| 0.0 | 18.5 | 6068 | 0.0000 |
| 0.0 | 18.75 | 6150 | 0.0000 |
| 0.0 | 19.0 | 6232 | 0.0000 |
| 0.0 | 19.25 | 6314 | 0.0000 |
| 0.0 | 19.5 | 6396 | 0.0000 |
| 0.0 | 19.75 | 6478 | 0.0000 |
| 0.0 | 20.0 | 6560 | 0.0000 |
| 0.0 | 20.25 | 6642 | 0.0000 |
| 0.0 | 20.5 | 6724 | 0.0000 |
| 0.0 | 20.75 | 6806 | 0.0000 |
| 0.0 | 21.0 | 6888 | 0.0000 |
| 0.0 | 21.25 | 6970 | 0.0000 |
| 0.0 | 21.5 | 7052 | 0.0000 |
| 0.0 | 21.75 | 7134 | 0.0000 |
| 0.0 | 22.0 | 7216 | 0.0000 |
| 0.0 | 22.25 | 7298 | 0.0000 |
| 0.0 | 22.5 | 7380 | 0.0000 |
| 0.0 | 22.75 | 7462 | 0.0000 |
| 0.0 | 23.0 | 7544 | 0.0000 |
| 0.0 | 23.25 | 7626 | 0.0000 |
| 0.0 | 23.5 | 7708 | 0.0000 |
| 0.0 | 23.75 | 7790 | 0.0000 |
| 0.0 | 24.0 | 7872 | 0.0000 |
| 0.0 | 24.25 | 7954 | 0.0000 |
| 0.0 | 24.5 | 8036 | 0.0000 |
| 0.0 | 24.75 | 8118 | 0.0000 |
| 0.0 | 25.0 | 8200 | 0.0000 |
| 0.0 | 25.25 | 8282 | 0.0000 |
| 0.0 | 25.5 | 8364 | 0.0000 |
| 0.0 | 25.75 | 8446 | 0.0000 |
| 0.0 | 26.0 | 8528 | 0.0000 |
| 0.0 | 26.25 | 8610 | 0.0000 |
| 0.0 | 26.5 | 8692 | 0.0000 |
| 0.0 | 26.75 | 8774 | 0.0000 |
| 0.0 | 27.0 | 8856 | 0.0000 |
| 0.0 | 27.25 | 8938 | 0.0000 |
| 0.0 | 27.5 | 9020 | 0.0000 |
| 0.0 | 27.75 | 9102 | 0.0000 |
| 0.0 | 28.0 | 9184 | 0.0000 |
| 0.0 | 28.25 | 9266 | 0.0000 |
| 0.0 | 28.5 | 9348 | 0.0000 |
| 0.0 | 28.75 | 9430 | 0.0000 |
| 0.0 | 29.0 | 9512 | 0.0000 |
| 0.0 | 29.25 | 9594 | 0.0000 |
| 0.0 | 29.5 | 9676 | 0.0000 |
| 0.0 | 29.75 | 9758 | 0.0000 |
| 0.0 | 30.0 | 9840 | 0.0000 |
| 0.0 | 30.25 | 9922 | 0.0000 |
| 0.0 | 30.5 | 10004 | 0.0000 |
| 0.0 | 30.75 | 10086 | 0.0000 |
| 0.0 | 31.0 | 10168 | 0.0000 |
| 0.0 | 31.25 | 10250 | 0.0000 |
| 0.0 | 31.5 | 10332 | 0.0000 |
| 0.0 | 31.75 | 10414 | 0.0000 |
| 0.0 | 32.0 | 10496 | 0.0000 |
| 0.0 | 32.25 | 10578 | 0.0000 |
| 0.0 | 32.5 | 10660 | 0.0000 |
| 0.0 | 32.75 | 10742 | 0.0000 |
| 0.0 | 33.0 | 10824 | 0.0000 |
| 0.0 | 33.25 | 10906 | 0.0000 |
| 0.0 | 33.5 | 10988 | 0.0000 |
| 0.0 | 33.75 | 11070 | 0.0000 |
| 0.0 | 34.0 | 11152 | 0.0000 |
| 0.0 | 34.25 | 11234 | 0.0000 |
| 0.0 | 34.5 | 11316 | 0.0000 |
| 0.0 | 34.75 | 11398 | 0.0000 |
| 0.0 | 35.0 | 11480 | 0.0000 |
| 0.0 | 35.25 | 11562 | 0.0000 |
| 0.0 | 35.5 | 11644 | 0.0000 |
| 0.0 | 35.75 | 11726 | 0.0000 |
| 0.0 | 36.0 | 11808 | 0.0000 |
| 0.0 | 36.25 | 11890 | 0.0000 |
| 0.0 | 36.5 | 11972 | 0.0000 |
| 0.0 | 36.75 | 12054 | 0.0000 |
| 0.0 | 37.0 | 12136 | 0.0000 |
| 0.0 | 37.25 | 12218 | 0.0000 |
| 0.0 | 37.5 | 12300 | 0.0000 |
| 0.0 | 37.75 | 12382 | 0.0000 |
| 0.0 | 38.0 | 12464 | 0.0000 |
| 0.0 | 38.25 | 12546 | 0.0000 |
| 0.0 | 38.5 | 12628 | 0.0000 |
| 0.0 | 38.75 | 12710 | 0.0000 |
| 0.0 | 39.0 | 12792 | 0.0000 |
| 0.0 | 39.25 | 12874 | 0.0000 |
| 0.0 | 39.5 | 12956 | 0.0000 |
| 0.0 | 39.75 | 13038 | 0.0000 |
| 0.0 | 40.0 | 13120 | 0.0000 |
| 0.0 | 40.25 | 13202 | 0.0000 |
| 0.0 | 40.5 | 13284 | 0.0000 |
| 0.0 | 40.75 | 13366 | 0.0000 |
| 0.0 | 41.0 | 13448 | 0.0000 |
| 0.0 | 41.25 | 13530 | 0.0000 |
| 0.0 | 41.5 | 13612 | 0.0000 |
| 0.0 | 41.75 | 13694 | 0.0000 |
| 0.0 | 42.0 | 13776 | 0.0000 |
| 0.0 | 42.25 | 13858 | 0.0000 |
| 0.0 | 42.5 | 13940 | 0.0000 |
| 0.0 | 42.75 | 14022 | 0.0000 |
| 0.0 | 43.0 | 14104 | 0.0000 |
| 0.0 | 43.25 | 14186 | 0.0000 |
| 0.0 | 43.5 | 14268 | 0.0000 |
| 0.0 | 43.75 | 14350 | 0.0000 |
| 0.0 | 44.0 | 14432 | 0.0000 |
| 0.0 | 44.25 | 14514 | 0.0000 |
| 0.0 | 44.5 | 14596 | 0.0000 |
| 0.0 | 44.75 | 14678 | 0.0000 |
| 0.0 | 45.0 | 14760 | 0.0000 |
| 0.0 | 45.25 | 14842 | 0.0000 |
| 0.0 | 45.5 | 14924 | 0.0000 |
| 0.0 | 45.75 | 15006 | 0.0000 |
| 0.0 | 46.0 | 15088 | 0.0000 |
| 0.0 | 46.25 | 15170 | 0.0000 |
| 0.0 | 46.5 | 15252 | 0.0000 |
| 0.0 | 46.75 | 15334 | 0.0000 |
| 0.0 | 47.0 | 15416 | 0.0000 |
| 0.0 | 47.25 | 15498 | 0.0000 |
| 0.0 | 47.5 | 15580 | 0.0000 |
| 0.0 | 47.75 | 15662 | 0.0000 |
| 0.0 | 48.0 | 15744 | 0.0000 |
| 0.0 | 48.25 | 15826 | 0.0000 |
| 0.0 | 48.5 | 15908 | 0.0000 |
| 0.0 | 48.75 | 15990 | 0.0000 |
| 0.0 | 49.0 | 16072 | 0.0000 |
| 0.0 | 49.25 | 16154 | 0.0000 |
| 0.0 | 49.5 | 16236 | 0.0000 |
| 0.0 | 49.75 | 16318 | 0.0000 |
| 0.0 | 50.0 | 16400 | 0.0000 |
### Framework versions
- PEFT 0.11.1
- Transformers 4.41.1
- Pytorch 2.1.2+cu118
- Datasets 2.19.1
- Tokenizers 0.19.1