mistral-rand

This model is a fine-tuned version of TheBloke/Mistral-7B-v0.1-GPTQ on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4471

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 15
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.7543	0.03	50	0.9190
0.8445	0.05	100	0.7860
0.7819	0.07	150	0.7460
0.7231	0.1	200	0.7147
0.6985	0.12	250	0.6924
0.6887	0.15	300	0.6823
0.6836	0.17	350	0.6702
0.6624	0.2	400	0.6574
0.6712	0.23	450	0.6507
0.6354	0.25	500	0.6417
0.6089	0.28	550	0.6373
0.6236	0.3	600	0.6284
0.6161	0.33	650	0.6228
0.6367	0.35	700	0.6152
0.6329	0.38	750	0.6097
0.5944	0.4	800	0.6076
0.6036	0.42	850	0.6030
0.5767	0.45	900	0.5989
0.6079	0.47	950	0.5954
0.5915	0.5	1000	0.5916
0.5911	0.53	1050	0.5859
0.5752	0.55	1100	0.5847
0.5698	0.57	1150	0.5802
0.5813	0.6	1200	0.5754
0.5918	0.62	1250	0.5735
0.5587	0.65	1300	0.5677
0.5933	0.68	1350	0.5620
0.5262	0.7	1400	0.5522
0.5455	0.72	1450	0.5457
0.5472	0.75	1500	0.5416
0.536	0.78	1550	0.5400
0.527	0.8	1600	0.5393
0.5516	0.82	1650	0.5350
0.5578	0.85	1700	0.5356
0.5501	0.88	1750	0.5297
0.5316	0.9	1800	0.5288
0.5436	0.93	1850	0.5268
0.514	0.95	1900	0.5295
0.5249	0.97	1950	0.5246
0.538	1.0	2000	0.5226
0.4967	1.02	2050	0.5237
0.4991	1.05	2100	0.5261
0.5142	1.07	2150	0.5203
0.4891	1.1	2200	0.5174
0.5058	1.12	2250	0.5173
0.4895	1.15	2300	0.5182
0.4918	1.18	2350	0.5139
0.485	1.2	2400	0.5091
0.5173	1.23	2450	0.5121
0.5021	1.25	2500	0.5116
0.4834	1.27	2550	0.5097
0.4754	1.3	2600	0.5137
0.4907	1.32	2650	0.5059
0.5155	1.35	2700	0.5051
0.4965	1.38	2750	0.5050
0.5148	1.4	2800	0.5043
0.4709	1.43	2850	0.5032
0.4864	1.45	2900	0.5037
0.4794	1.48	2950	0.5029
0.4803	1.5	3000	0.5012
0.4843	1.52	3050	0.5017
0.4726	1.55	3100	0.4984
0.4773	1.57	3150	0.4968
0.4673	1.6	3200	0.4995
0.4803	1.62	3250	0.4990
0.4926	1.65	3300	0.4965
0.4814	1.68	3350	0.4973
0.4714	1.7	3400	0.4930
0.4797	1.73	3450	0.4903
0.4807	1.75	3500	0.4932
0.4815	1.77	3550	0.4888
0.4852	1.8	3600	0.4874
0.4802	1.82	3650	0.4887
0.4701	1.85	3700	0.4897
0.4572	1.88	3750	0.4873
0.4469	1.9	3800	0.4878
0.478	1.93	3850	0.4885
0.4449	1.95	3900	0.4866
0.4634	1.98	3950	0.4843
0.4718	2.0	4000	0.4838
0.4458	2.02	4050	0.4822
0.461	2.05	4100	0.4801
0.4247	2.08	4150	0.4856
0.4325	2.1	4200	0.4830
0.4354	2.12	4250	0.4827
0.4313	2.15	4300	0.4807
0.4753	2.17	4350	0.4812
0.4442	2.2	4400	0.4833
0.4431	2.23	4450	0.4851
0.4485	2.25	4500	0.4815
0.4416	2.27	4550	0.4813
0.4613	2.3	4600	0.4777
0.4121	2.33	4650	0.4775
0.4311	2.35	4700	0.4768
0.4532	2.38	4750	0.4765
0.4342	2.4	4800	0.4781
0.4189	2.42	4850	0.4743
0.443	2.45	4900	0.4742
0.4596	2.48	4950	0.4734
0.4193	2.5	5000	0.4719
0.4321	2.52	5050	0.4723
0.4456	2.55	5100	0.4713
0.4464	2.58	5150	0.4694
0.4273	2.6	5200	0.4700
0.4239	2.62	5250	0.4701
0.4282	2.65	5300	0.4687
0.4303	2.67	5350	0.4686
0.4559	2.7	5400	0.4695
0.4542	2.73	5450	0.4692
0.4532	2.75	5500	0.4685
0.4505	2.77	5550	0.4663
0.4533	2.8	5600	0.4660
0.4351	2.83	5650	0.4640
0.4354	2.85	5700	0.4651
0.4374	2.88	5750	0.4664
0.4571	2.9	5800	0.4662
0.4663	2.92	5850	0.4636
0.4211	2.95	5900	0.4645
0.4349	2.98	5950	0.4622
0.4167	3.0	6000	0.4634
0.4176	3.02	6050	0.4621
0.4387	3.05	6100	0.4607
0.395	3.08	6150	0.4638
0.4186	3.1	6200	0.4623
0.3993	3.12	6250	0.4622
0.4009	3.15	6300	0.4631
0.4033	3.17	6350	0.4640
0.389	3.2	6400	0.4662
0.4037	3.23	6450	0.4618
0.4287	3.25	6500	0.4617
0.3917	3.27	6550	0.4611
0.3944	3.3	6600	0.4626
0.4088	3.33	6650	0.4622
0.4205	3.35	6700	0.4604
0.4273	3.38	6750	0.4608
0.4139	3.4	6800	0.4607
0.3888	3.42	6850	0.4603
0.4353	3.45	6900	0.4573
0.4222	3.48	6950	0.4577
0.4083	3.5	7000	0.4571
0.4161	3.52	7050	0.4560
0.3879	3.55	7100	0.4540
0.3819	3.58	7150	0.4570
0.4345	3.6	7200	0.4551
0.4101	3.62	7250	0.4569
0.4194	3.65	7300	0.4543
0.4066	3.67	7350	0.4563
0.4144	3.7	7400	0.4553
0.4134	3.73	7450	0.4566
0.3906	3.75	7500	0.4550
0.4128	3.77	7550	0.4546
0.4227	3.8	7600	0.4535
0.4069	3.83	7650	0.4517
0.3927	3.85	7700	0.4548
0.3977	3.88	7750	0.4521
0.4184	3.9	7800	0.4516
0.3854	3.92	7850	0.4513
0.4129	3.95	7900	0.4524
0.3998	3.98	7950	0.4548
0.4227	4.0	8000	0.4534
0.3788	4.03	8050	0.4520
0.3732	4.05	8100	0.4501
0.375	4.08	8150	0.4565
0.3845	4.1	8200	0.4515
0.378	4.12	8250	0.4492
0.3874	4.15	8300	0.4508
0.3802	4.17	8350	0.4510
0.3596	4.2	8400	0.4524
0.4009	4.22	8450	0.4549
0.4105	4.25	8500	0.4515
0.3716	4.28	8550	0.4508
0.3673	4.3	8600	0.4497
0.3882	4.33	8650	0.4513
0.375	4.35	8700	0.4524
0.3654	4.38	8750	0.4503
0.3983	4.4	8800	0.4509
0.4067	4.42	8850	0.4487
0.3966	4.45	8900	0.4519
0.378	4.47	8950	0.4505
0.3755	4.5	9000	0.4508
0.3855	4.53	9050	0.4500
0.3938	4.55	9100	0.4527
0.3946	4.58	9150	0.4531
0.3752	4.6	9200	0.4506
0.3723	4.62	9250	0.4459
0.3704	4.65	9300	0.4467
0.3861	4.67	9350	0.4484
0.3965	4.7	9400	0.4481
0.3972	4.72	9450	0.4482
0.3917	4.75	9500	0.4447
0.3688	4.78	9550	0.4473
0.3861	4.8	9600	0.4491
0.3593	4.83	9650	0.4491
0.3916	4.85	9700	0.4432
0.3748	4.88	9750	0.4432
0.3921	4.9	9800	0.4459
0.3745	4.92	9850	0.4457
0.4002	4.95	9900	0.4443
0.3767	4.97	9950	0.4430
0.3537	5.0	10000	0.4470
0.3673	5.03	10050	0.4531
0.3506	5.05	10100	0.4474
0.3506	5.08	10150	0.4497
0.3622	5.1	10200	0.4471

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.16.1
• Tokenizers 0.15.0