Mistral-7B-v0.3_pct_ortho_r32

This model is a fine-tuned version of mistralai/Mistral-7B-v0.3 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.9721

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 64
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.02
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.9673	0.0206	8	1.9675
1.982	0.0413	16	1.9762
1.9563	0.0619	24	1.9745
1.957	0.0825	32	1.9779
2.0248	0.1032	40	1.9852
1.9753	0.1238	48	1.9988
1.9752	0.1444	56	1.9974
2.0253	0.1651	64	1.9947
2.0073	0.1857	72	1.9872
1.9826	0.2063	80	1.9953
1.9907	0.2270	88	2.0015
1.9795	0.2476	96	1.9951
1.9882	0.2682	104	2.0020
1.9896	0.2889	112	1.9963
2.0177	0.3095	120	2.0146
2.0131	0.3301	128	2.0013
2.0384	0.3508	136	2.0017
2.0587	0.3714	144	2.0019
1.9998	0.3920	152	1.9965
1.9729	0.4127	160	1.9905
2.0339	0.4333	168	2.0233
2.0029	0.4539	176	1.9972
1.997	0.4746	184	1.9976
1.9808	0.4952	192	2.0007
2.0169	0.5158	200	1.9872
1.9605	0.5364	208	1.9975
2.0195	0.5571	216	1.9963
1.9619	0.5777	224	1.9878
1.9361	0.5983	232	2.0045
1.9932	0.6190	240	1.9815
1.9519	0.6396	248	1.9896
1.9843	0.6602	256	1.9901
1.963	0.6809	264	1.9820
1.9376	0.7015	272	1.9793
1.9876	0.7221	280	1.9885
2.0157	0.7428	288	1.9834
2.011	0.7634	296	1.9843
2.0179	0.7840	304	1.9779
1.9693	0.8047	312	1.9787
1.9632	0.8253	320	1.9824
1.9367	0.8459	328	1.9776
1.9824	0.8666	336	1.9730
1.9911	0.8872	344	1.9719
2.0075	0.9078	352	1.9730
1.9809	0.9285	360	1.9730
1.9971	0.9491	368	1.9722
1.9913	0.9697	376	1.9720
1.916	0.9904	384	1.9721

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.3.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1