WeniGPT-Agents-Mistral-1.0.0-SFT-1.0.14-DPO

This model is a fine-tuned version of Weni/WeniGPT-Agents-Mistral-1.0.0-SFT-merged on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1709
• Rewards/chosen: 1.9941
• Rewards/rejected: -0.4449
• Rewards/accuracies: 0.8571
• Rewards/margins: 2.4390
• Logps/rejected: -161.0436
• Logps/chosen: -111.4245
• Logits/rejected: -1.8499
• Logits/chosen: -1.8319

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 180
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.5263	0.9677	30	0.5183	0.3988	-0.0166	0.7143	0.4154	-159.6158	-116.7421	-1.8403	-1.8221
0.2814	1.9355	60	0.3516	0.9688	-0.0208	0.7143	0.9896	-159.6299	-114.8421	-1.8443	-1.8259
0.1778	2.9032	90	0.2655	1.3864	-0.0997	0.8571	1.4861	-159.8928	-113.4503	-1.8470	-1.8286
0.1388	3.8710	120	0.2128	1.7020	-0.2501	0.8571	1.9521	-160.3941	-112.3981	-1.8494	-1.8311
0.1349	4.8387	150	0.1841	1.9322	-0.3766	0.8571	2.3088	-160.8158	-111.6308	-1.8499	-1.8319
0.1178	5.8065	180	0.1709	1.9941	-0.4449	0.8571	2.4390	-161.0436	-111.4245	-1.8499	-1.8319

Framework versions

• PEFT 0.10.0
• Transformers 4.40.0
• Pytorch 2.1.0+cu118
• Datasets 2.18.0
• Tokenizers 0.19.1