CHAFIK12/second_sql_converter_model

fine_tuning_codellama

This model is a fine-tuned version of codellama/CodeLlama-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1513

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• _load_in_8bit: True
• _load_in_4bit: False
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: fp4
• bnb_4bit_use_double_quant: False
• bnb_4bit_compute_dtype: float32
• bnb_4bit_quant_storage: uint8
• load_in_4bit: False
• load_in_8bit: True

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• training_steps: 400
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.1085	0.1010	20	1.0942
0.729	0.2020	40	0.5856
0.4941	0.3030	60	0.3775
0.2508	0.4040	80	0.2868
0.2139	0.5051	100	0.2528
0.2612	0.6061	120	0.2115
0.154	0.7071	140	0.2073
0.2558	0.8081	160	0.1936
0.1553	0.9091	180	0.1832
0.1831	1.0101	200	0.1872
0.2135	1.1111	220	0.1729
0.1278	1.2121	240	0.1724
0.21	1.3131	260	0.1676
0.1328	1.4141	280	0.1619
0.1731	1.5152	300	0.1651
0.1668	1.6162	320	0.1577
0.1259	1.7172	340	0.1565
0.191	1.8182	360	0.1543
0.129	1.9192	380	0.1515
0.1693	2.0202	400	0.1513

Framework versions

• PEFT 0.6.0.dev0
• Transformers 4.42.0.dev0
• Pytorch 2.1.2
• Datasets 2.19.2
• Tokenizers 0.19.1