GaetanMichelet's picture
End of training
bc1938d verified
metadata
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
datasets:
  - GaetanMichelet/chat-60_ft_task-2
library_name: peft
license: llama3.1
tags:
  - alignment-handbook
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: Llama-31-8B_task-2_60-samples_config-3_full
    results: []

Llama-31-8B_task-2_60-samples_config-3_full

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2 dataset. It achieves the following results on the evaluation set:

  • Loss: 1.0710

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 1
  • eval_batch_size: 1
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 150

Training results

Training Loss Epoch Step Validation Loss
1.5659 0.8696 5 1.5846
1.5947 1.9130 11 1.5811
1.6305 2.9565 17 1.5758
1.5682 4.0 23 1.5673
1.5687 4.8696 28 1.5571
1.5556 5.9130 34 1.5406
1.4699 6.9565 40 1.5190
1.5027 8.0 46 1.4958
1.5203 8.8696 51 1.4719
1.4872 9.9130 57 1.4445
1.4184 10.9565 63 1.4151
1.3817 12.0 69 1.3868
1.3397 12.8696 74 1.3648
1.3234 13.9130 80 1.3390
1.2893 14.9565 86 1.3122
1.2999 16.0 92 1.2852
1.2212 16.8696 97 1.2628
1.234 17.9130 103 1.2358
1.1704 18.9565 109 1.2078
1.1499 20.0 115 1.1796
1.1265 20.8696 120 1.1570
1.0716 21.9130 126 1.1357
1.0332 22.9565 132 1.1223
1.0631 24.0 138 1.1155
1.0659 24.8696 143 1.1111
1.0637 25.9130 149 1.1068
0.9979 26.9565 155 1.1031
1.0495 28.0 161 1.0993
1.0126 28.8696 166 1.0966
0.9884 29.9130 172 1.0938
1.0366 30.9565 178 1.0909
1.0434 32.0 184 1.0886
1.0222 32.8696 189 1.0862
0.9978 33.9130 195 1.0842
0.9593 34.9565 201 1.0824
1.0383 36.0 207 1.0804
0.9958 36.8696 212 1.0792
0.9774 37.9130 218 1.0778
0.9853 38.9565 224 1.0763
0.9241 40.0 230 1.0747
1.0387 40.8696 235 1.0743
0.9616 41.9130 241 1.0733
0.9909 42.9565 247 1.0724
0.9055 44.0 253 1.0720
1.0025 44.8696 258 1.0722
0.9325 45.9130 264 1.0711
0.8921 46.9565 270 1.0723
0.9079 48.0 276 1.0710
0.9615 48.8696 281 1.0729
0.9517 49.9130 287 1.0718
0.8619 50.9565 293 1.0730
0.8894 52.0 299 1.0739
0.8389 52.8696 304 1.0742
0.9032 53.9130 310 1.0750
0.9015 54.9565 316 1.0760

Framework versions

  • PEFT 0.12.0
  • Transformers 4.44.0
  • Pytorch 2.1.2+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1