sft-gemma-2-2b-ultrafeedback-binarized-20240920-114857

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2499

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1.41e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 128
• total_train_batch_size: 1024
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.3539	0.3350	20	1.3197
1.3073	0.6700	40	1.2901
1.307	1.0050	60	1.2739
1.2327	1.3400	80	1.2642
1.2515	1.6750	100	1.2567
1.2139	2.0099	120	1.2523
1.2023	2.3449	140	1.2509
1.2031	2.6799	160	1.2499

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1