w2v-bert-2.0-seeyuh

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4176
• Wer: 0.1450

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 10000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.5769	0.7524	500	0.5834	0.4912
0.3206	1.5049	1000	0.4060	0.3510
0.2663	2.2573	1500	0.3600	0.2888
0.2271	3.0098	2000	0.3411	0.2574
0.1823	3.7622	2500	0.3274	0.2485
0.15	4.5147	3000	0.3120	0.2219
0.1328	5.2671	3500	0.2971	0.2033
0.1354	6.0196	4000	0.2908	0.1974
0.1186	6.7720	4500	0.2875	0.1917
0.0617	7.5245	5000	0.3074	0.1832
0.0673	8.2769	5500	0.3146	0.1790
0.0882	9.0293	6000	0.3023	0.1687
0.0622	9.7818	6500	0.3038	0.1651
0.0398	10.5342	7000	0.3230	0.1672
0.027	11.2867	7500	0.3674	0.1578
0.0316	12.0391	8000	0.3585	0.1542
0.0271	12.7916	8500	0.3803	0.1499
0.0364	13.5440	9000	0.3918	0.1496
0.0047	14.2965	9500	0.4113	0.1465
0.0101	15.0489	10000	0.4176	0.1450

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu124
• Datasets 2.20.0
• Tokenizers 0.19.1