G0514HMA10H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.7415

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.0338	0.09	10	0.3604
-0.372	0.18	20	-1.5506
-2.7334	0.27	30	-4.3453
-5.5531	0.36	40	-7.2099
-8.3571	0.45	50	-9.9832
-11.0034	0.54	60	-12.4742
-13.3507	0.63	70	-14.4048
-15.0045	0.73	80	-15.7395
-16.1247	0.82	90	-16.6047
-16.8322	0.91	100	-17.0754
-17.1905	1.0	110	-17.3142
-17.3786	1.09	120	-17.4479
-17.4861	1.18	130	-17.5251
-17.5408	1.27	140	-17.5761
-17.5877	1.36	150	-17.6039
-17.6148	1.45	160	-17.6306
-17.6334	1.54	170	-17.6459
-17.6562	1.63	180	-17.6606
-17.6709	1.72	190	-17.6791
-17.6849	1.81	200	-17.6912
-17.6955	1.9	210	-17.7036
-17.7055	1.99	220	-17.7137
-17.7159	2.08	230	-17.7203
-17.7265	2.18	240	-17.7251
-17.7262	2.27	250	-17.7294
-17.7319	2.36	260	-17.7344
-17.7389	2.45	270	-17.7360
-17.7395	2.54	280	-17.7393
-17.7427	2.63	290	-17.7389
-17.7449	2.72	300	-17.7409
-17.7413	2.81	310	-17.7415
-17.7452	2.9	320	-17.7415
-17.7458	2.99	330	-17.7415

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0