adapter_head_full_const_lr_1e-4_l20-l23_const_lr_1e-7_l1-l19
This model is a fine-tuned version of facebook/w2v-bert-2.0 on the common_voice_17_0 dataset. It achieves the following results on the evaluation set:
- Loss: 0.3638
- Wer: 0.1946
- Cer: 0.0323
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 100
- mixed_precision_training: Native AMP
Training results
Training Loss | Epoch | Step | Validation Loss | Wer | Cer |
---|---|---|---|---|---|
0.1716 | 2.3077 | 750 | 0.2372 | 0.3536 | 0.0576 |
0.0888 | 4.6154 | 1500 | 0.2341 | 0.3066 | 0.0509 |
0.0487 | 6.9231 | 2250 | 0.2555 | 0.2823 | 0.0467 |
0.0221 | 9.2308 | 3000 | 0.2957 | 0.2668 | 0.0444 |
0.0193 | 11.5385 | 3750 | 0.3013 | 0.2461 | 0.0411 |
0.0162 | 13.8462 | 4500 | 0.3230 | 0.2584 | 0.0431 |
0.0107 | 16.1538 | 5250 | 0.3377 | 0.2454 | 0.0408 |
0.0106 | 18.4615 | 6000 | 0.3370 | 0.2473 | 0.0413 |
0.0111 | 20.7692 | 6750 | 0.3457 | 0.2448 | 0.0414 |
0.0084 | 23.0769 | 7500 | 0.3279 | 0.2302 | 0.0387 |
0.0083 | 25.3846 | 8250 | 0.3402 | 0.2308 | 0.0382 |
0.009 | 27.6923 | 9000 | 0.3411 | 0.2302 | 0.0384 |
0.0085 | 30.0 | 9750 | 0.3311 | 0.2292 | 0.0375 |
0.006 | 32.3077 | 10500 | 0.3492 | 0.2238 | 0.0371 |
0.0063 | 34.6154 | 11250 | 0.3560 | 0.2330 | 0.0381 |
0.0064 | 36.9231 | 12000 | 0.3584 | 0.2259 | 0.0379 |
0.0054 | 39.2308 | 12750 | 0.3484 | 0.2123 | 0.0351 |
0.0041 | 41.5385 | 13500 | 0.3565 | 0.2131 | 0.0356 |
0.0044 | 43.8462 | 14250 | 0.3522 | 0.2171 | 0.0363 |
0.0025 | 46.1538 | 15000 | 0.3702 | 0.2084 | 0.0350 |
0.0073 | 48.4615 | 15750 | 0.3579 | 0.2203 | 0.0360 |
0.0048 | 50.7692 | 16500 | 0.3462 | 0.2116 | 0.0353 |
0.0053 | 53.0769 | 17250 | 0.3264 | 0.2014 | 0.0337 |
0.0028 | 55.3846 | 18000 | 0.3560 | 0.2059 | 0.0343 |
0.0039 | 57.6923 | 18750 | 0.3685 | 0.2081 | 0.0348 |
0.0026 | 60.0 | 19500 | 0.3649 | 0.2075 | 0.0347 |
0.0027 | 62.3077 | 20250 | 0.3636 | 0.2091 | 0.0350 |
0.0038 | 64.6154 | 21000 | 0.3675 | 0.2147 | 0.0350 |
0.0024 | 66.9231 | 21750 | 0.3707 | 0.2050 | 0.0341 |
0.0045 | 69.2308 | 22500 | 0.3397 | 0.1961 | 0.0329 |
0.0032 | 71.5385 | 23250 | 0.3645 | 0.1985 | 0.0332 |
0.0041 | 73.8462 | 24000 | 0.3451 | 0.2047 | 0.0338 |
0.0018 | 76.1538 | 24750 | 0.3468 | 0.1935 | 0.0321 |
0.0045 | 78.4615 | 25500 | 0.3366 | 0.1982 | 0.0332 |
0.0023 | 80.7692 | 26250 | 0.3551 | 0.1996 | 0.0336 |
0.0022 | 83.0769 | 27000 | 0.3778 | 0.1948 | 0.0331 |
0.0026 | 85.3846 | 27750 | 0.3622 | 0.1950 | 0.0328 |
0.0013 | 87.6923 | 28500 | 0.3600 | 0.1908 | 0.0319 |
0.0032 | 90.0 | 29250 | 0.3632 | 0.1945 | 0.0324 |
0.0027 | 92.3077 | 30000 | 0.3436 | 0.1913 | 0.0320 |
0.002 | 94.6154 | 30750 | 0.3721 | 0.1985 | 0.0334 |
0.0022 | 96.9231 | 31500 | 0.3659 | 0.1966 | 0.0330 |
0.0025 | 99.2308 | 32250 | 0.3638 | 0.1946 | 0.0323 |
Framework versions
- Transformers 4.41.2
- Pytorch 2.3.0+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1
- Downloads last month
- 0
Inference API (serverless) is not available, repository is disabled.
Model tree for LevonHakobyan/adapter_head_full_const_lr_1e-4_l20-l23_const_lr_1e-7_l1-l19
Base model
facebook/w2v-bert-2.0
Finetuned
this model