phi_1_5_dpo_ep6 / README.md
hushell's picture
init (#1)
f70168c verified
metadata
base_model: microsoft/phi-1_5
tags:
  - alignment-handbook
  - generated_from_trainer
datasets:
  - argilla/ultrafeedback-binarized-preferences-cleaned
model-index:
  - name: phi_1_5_dpo_ep6
    results: []

phi_1_5_dpo_ep6

This model is a fine-tuned version of /home/work/saic-llm-2023/checkpoints/microsoft/phi-1_5 on the argilla/ultrafeedback-binarized-preferences-cleaned dataset. It achieves the following results on the evaluation set:

  • Loss: 0.4748
  • Rewards/chosen: -0.9135
  • Rewards/rejected: -1.9448
  • Rewards/accuracies: 0.7937
  • Rewards/margins: 1.0313
  • Logps/rejected: -618.5530
  • Logps/chosen: -634.6866
  • Logits/rejected: 3.4318
  • Logits/chosen: 3.4052

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-07
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 64
  • total_eval_batch_size: 32
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 6

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.6881 0.11 100 0.6856 0.0468 0.0298 0.7024 0.0170 -421.0949 -538.6564 4.8883 4.6646
0.6692 0.22 200 0.6642 0.1742 0.0988 0.7123 0.0754 -414.1955 -525.9189 4.8718 4.6370
0.6368 0.33 300 0.6442 0.2557 0.1261 0.7083 0.1296 -411.4657 -517.7680 4.8407 4.5968
0.6283 0.43 400 0.6283 0.2608 0.0812 0.7083 0.1795 -415.9522 -517.2609 4.7629 4.5156
0.6052 0.54 500 0.6132 0.1429 -0.0998 0.7103 0.2427 -434.0545 -529.0491 4.5516 4.3153
0.5923 0.65 600 0.6008 0.1425 -0.1628 0.7123 0.3053 -440.3539 -529.0887 4.4588 4.2289
0.5899 0.76 700 0.5880 0.0755 -0.2915 0.7083 0.3670 -453.2271 -535.7857 4.3444 4.1349
0.558 0.87 800 0.5715 -0.0965 -0.5304 0.7262 0.4339 -477.1144 -552.9822 4.2704 4.0642
0.5495 0.98 900 0.5552 -0.2658 -0.7677 0.7341 0.5019 -500.8484 -569.9210 4.1976 4.0015
0.5124 1.09 1000 0.5473 -0.3871 -0.9394 0.7321 0.5523 -518.0129 -582.0427 4.0959 3.9125
0.5322 1.19 1100 0.5400 -0.3641 -0.9463 0.7579 0.5821 -518.7011 -579.7518 4.0436 3.8715
0.5281 1.3 1200 0.5344 -0.5340 -1.1498 0.7460 0.6158 -539.0579 -596.7365 3.9368 3.7842
0.5063 1.41 1300 0.5297 -0.3754 -0.9975 0.7579 0.6221 -523.8221 -580.8731 4.0135 3.8499
0.5073 1.52 1400 0.5216 -0.3819 -1.0300 0.7758 0.6481 -527.0738 -581.5236 3.9401 3.7846
0.5156 1.63 1500 0.5177 -0.5748 -1.2824 0.7560 0.7077 -552.3166 -600.8123 3.7868 3.6678
0.5072 1.74 1600 0.5138 -0.4973 -1.2122 0.7798 0.7149 -545.2914 -593.0637 3.7791 3.6614
0.4908 1.85 1700 0.5077 -0.5479 -1.2972 0.7798 0.7493 -553.7918 -598.1292 3.7893 3.6696
0.5109 1.95 1800 0.5068 -0.6157 -1.3930 0.7758 0.7773 -563.3733 -604.9089 3.7679 3.6556
0.4779 2.06 1900 0.5005 -0.6247 -1.4169 0.7738 0.7922 -565.7673 -605.8088 3.7118 3.6062
0.4833 2.17 2000 0.4992 -0.6841 -1.5026 0.7698 0.8185 -574.3334 -611.7432 3.6739 3.5849
0.4879 2.28 2100 0.4967 -0.8128 -1.6654 0.7698 0.8526 -590.6146 -624.6127 3.5692 3.5030
0.4645 2.39 2200 0.4927 -0.6969 -1.5365 0.7857 0.8396 -577.7230 -613.0289 3.6647 3.5772
0.4587 2.5 2300 0.4936 -0.6024 -1.4533 0.7778 0.8509 -569.4068 -603.5743 3.6615 3.5790
0.437 2.61 2400 0.4921 -0.8826 -1.7724 0.7738 0.8897 -601.3099 -631.5984 3.4903 3.4343
0.4204 2.71 2500 0.4890 -0.8338 -1.7338 0.7758 0.8999 -597.4498 -626.7175 3.5447 3.4804
0.467 2.82 2600 0.4865 -0.5910 -1.4516 0.7877 0.8606 -569.2333 -602.4326 3.5690 3.5000
0.458 2.93 2700 0.4861 -0.7666 -1.6726 0.7837 0.9059 -591.3298 -620.0014 3.5208 3.4579
0.462 3.04 2800 0.4844 -0.7109 -1.6145 0.7917 0.9037 -585.5269 -614.4227 3.5553 3.4954
0.4258 3.15 2900 0.4888 -0.9814 -1.9414 0.7817 0.9600 -618.2142 -641.4772 3.4761 3.4227
0.4219 3.26 3000 0.4856 -0.8858 -1.8323 0.7937 0.9465 -607.3071 -631.9181 3.4895 3.4362
0.4295 3.37 3100 0.4823 -0.8140 -1.7651 0.7976 0.9511 -600.5797 -624.7327 3.4880 3.4357
0.4268 3.47 3200 0.4800 -0.8592 -1.8282 0.7976 0.9690 -606.8929 -629.2567 3.4536 3.4126
0.4338 3.58 3300 0.4785 -0.8784 -1.8458 0.7956 0.9674 -608.6551 -631.1731 3.4471 3.4096
0.4297 3.69 3400 0.4774 -0.9026 -1.8929 0.7956 0.9903 -613.3634 -633.5962 3.4710 3.4326
0.4133 3.8 3500 0.4785 -0.9173 -1.9072 0.7937 0.9899 -614.7964 -635.0674 3.4610 3.4232
0.4275 3.91 3600 0.4794 -1.0209 -2.0380 0.7837 1.0171 -627.8748 -645.4227 3.4635 3.4227
0.4224 4.02 3700 0.4784 -0.9130 -1.9086 0.7937 0.9955 -614.9320 -634.6396 3.4812 3.4400
0.4101 4.13 3800 0.4773 -0.9474 -1.9571 0.7877 1.0097 -619.7819 -638.0772 3.4569 3.4225
0.4295 4.23 3900 0.4790 -0.9893 -2.0096 0.7956 1.0203 -625.0361 -642.2666 3.4290 3.3998
0.4162 4.34 4000 0.4769 -0.9682 -1.9897 0.7956 1.0215 -623.0465 -640.1562 3.4342 3.4040
0.425 4.45 4100 0.4759 -0.9553 -1.9788 0.7917 1.0236 -621.9555 -638.8621 3.4580 3.4237
0.4155 4.56 4200 0.4778 -1.0183 -2.0573 0.7917 1.0390 -629.8077 -645.1696 3.4277 3.3981
0.4311 4.67 4300 0.4765 -0.9712 -2.0065 0.7897 1.0353 -624.7266 -640.4598 3.4413 3.4107
0.41 4.78 4400 0.4768 -0.9764 -2.0101 0.7917 1.0337 -625.0818 -640.9733 3.4387 3.4081
0.4127 4.89 4500 0.4749 -0.9599 -1.9994 0.7937 1.0395 -624.0168 -639.3277 3.4453 3.4160
0.453 4.99 4600 0.4748 -0.9231 -1.9528 0.7917 1.0297 -619.3519 -635.6462 3.4444 3.4142
0.4035 5.1 4700 0.4754 -0.9561 -1.9965 0.7897 1.0403 -623.7211 -638.9504 3.4293 3.4019
0.4225 5.21 4800 0.4753 -0.9471 -1.9855 0.7877 1.0384 -622.6226 -638.0461 3.4359 3.4077
0.3941 5.32 4900 0.4754 -0.9579 -1.9978 0.7897 1.0400 -623.8593 -639.1230 3.4282 3.4012
0.4093 5.43 5000 0.4748 -0.9135 -1.9448 0.7937 1.0313 -618.5530 -634.6866 3.4318 3.4052
0.3902 5.54 5100 0.4754 -0.9457 -1.9815 0.7956 1.0358 -622.2274 -637.9056 3.4281 3.4014
0.3795 5.65 5200 0.4753 -0.9484 -1.9852 0.7897 1.0368 -622.5895 -638.1724 3.4253 3.3988
0.3915 5.75 5300 0.4754 -0.9571 -1.9957 0.7956 1.0386 -623.6450 -639.0427 3.4242 3.3979
0.4075 5.86 5400 0.4756 -0.9566 -1.9949 0.7877 1.0383 -623.5674 -638.9974 3.4221 3.3962
0.4293 5.97 5500 0.4756 -0.9571 -1.9948 0.7897 1.0377 -623.5548 -639.0446 3.4230 3.3964

Framework versions

  • Transformers 4.38.0
  • Pytorch 2.1.2+cu118
  • Datasets 2.17.1
  • Tokenizers 0.15.0