xls-r-1b-bem-genbed-all

This model is a fine-tuned version of facebook/wav2vec2-xls-r-1b on the GENBED - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.2172
• Wer: 0.7294

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 5.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
4.6827	0.2644	200	2.8347	1.0
1.0401	0.5288	400	0.5636	0.9410
0.4289	0.7931	600	0.4018	0.9029
0.3449	1.0575	800	0.3604	0.8771
0.2954	1.3219	1000	0.3389	0.8741
0.2719	1.5863	1200	0.2962	0.8439
0.2472	1.8506	1400	0.2701	0.8053
0.2093	2.1150	1600	0.2599	0.8285
0.1725	2.3794	1800	0.2534	0.8375
0.1675	2.6438	2000	0.2406	0.7691
0.1632	2.9081	2200	0.2309	0.7616
0.1295	3.1725	2400	0.2387	0.7557
0.1082	3.4369	2600	0.2275	0.7329
0.1059	3.7013	2800	0.2240	0.7329
0.1049	3.9656	3000	0.2172	0.7294
0.0657	4.2300	3200	0.2320	0.7220
0.059	4.4944	3400	0.2341	0.7215
0.0582	4.7588	3600	0.2316	0.7116

Framework versions

• Transformers 4.45.0.dev0
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1