pretrain_wo-cot_w-asd_gpt

This model is a fine-tuned version of kakaobrain/kogpt on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 5.2695

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
9.3854	0.1290	1	10.0019
9.2319	0.2581	2	9.7577
8.9195	0.3871	3	9.4655
8.7866	0.5161	4	9.1438
8.4714	0.6452	5	9.1438
8.548	0.7742	6	8.8258
8.1645	0.9032	7	8.5218
7.9288	1.0323	8	8.1624
7.4999	1.1613	9	7.8294
7.0631	1.2903	10	7.5127
6.9252	1.4194	11	7.1942
6.7075	1.5484	12	6.8620
6.4293	1.6774	13	6.5825
6.1982	1.8065	14	6.2854
5.9671	1.9355	15	6.0680
5.5963	2.0645	16	5.8336
5.4335	2.1935	17	5.6719
5.3723	2.3226	18	5.5392
5.3661	2.4516	19	5.4076
5.0969	2.5806	20	5.3410
5.0227	2.7097	21	5.2695

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1