metadata
tags:
- generated_from_trainer
model-index:
- name: deberta-base-en-wiki
results: []
deberta-base-en-wiki
This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.1310
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 256
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1250.0
- num_epochs: 10
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 6.4739 | 0.0504 | 1250 | 6.4351 |
| 3.4566 | 0.1009 | 2500 | 3.2703 |
| 2.7823 | 0.1513 | 3750 | 2.6563 |
| 2.5242 | 0.2018 | 5000 | 2.4202 |
| 2.3816 | 0.2522 | 6250 | 2.2763 |
| 2.2723 | 0.3027 | 7500 | 2.1715 |
| 2.182 | 0.3531 | 8750 | 2.0933 |
| 2.1325 | 0.4035 | 10000 | 2.0317 |
| 2.0584 | 0.4540 | 11250 | 1.9770 |
| 2.0333 | 0.5044 | 12500 | 1.9288 |
| 1.9898 | 0.5549 | 13750 | 1.8972 |
| 1.9574 | 0.6053 | 15000 | 1.8557 |
| 1.9184 | 0.6558 | 16250 | 1.8324 |
| 1.8899 | 0.7062 | 17500 | 1.8049 |
| 1.8909 | 0.7567 | 18750 | 1.7788 |
| 1.8371 | 0.8071 | 20000 | 1.7558 |
| 1.8343 | 0.8575 | 21250 | 1.7374 |
| 1.8341 | 0.9080 | 22500 | 1.7256 |
| 1.7976 | 0.9584 | 23750 | 1.7011 |
| 1.7777 | 1.0089 | 25000 | 1.6865 |
| 1.7523 | 1.0593 | 26250 | 1.6715 |
| 1.7476 | 1.1098 | 27500 | 1.6581 |
| 1.7291 | 1.1602 | 28750 | 1.6432 |
| 1.7108 | 1.2106 | 30000 | 1.6333 |
| 1.7195 | 1.2611 | 31250 | 1.6198 |
| 1.6969 | 1.3115 | 32500 | 1.6109 |
| 1.6927 | 1.3620 | 33750 | 1.5965 |
| 1.6818 | 1.4124 | 35000 | 1.5917 |
| 1.6647 | 1.4629 | 36250 | 1.5827 |
| 1.6635 | 1.5133 | 37500 | 1.5704 |
| 1.6561 | 1.5637 | 38750 | 1.5593 |
| 1.6404 | 1.6142 | 40000 | 1.5527 |
| 1.627 | 1.6646 | 41250 | 1.5470 |
| 1.6292 | 1.7151 | 42500 | 1.5391 |
| 1.6111 | 1.7655 | 43750 | 1.5288 |
| 1.6154 | 1.8160 | 45000 | 1.5217 |
| 1.5993 | 1.8664 | 46250 | 1.5191 |
| 1.6028 | 1.9168 | 47500 | 1.5077 |
| 1.5861 | 1.9673 | 48750 | 1.5019 |
| 1.5793 | 2.0177 | 50000 | 1.4954 |
| 1.5664 | 2.0682 | 51250 | 1.4887 |
| 1.5723 | 2.1186 | 52500 | 1.4839 |
| 1.5715 | 2.1691 | 53750 | 1.4786 |
| 1.5612 | 2.2195 | 55000 | 1.4757 |
| 1.5499 | 2.2700 | 56250 | 1.4648 |
| 1.5542 | 2.3204 | 57500 | 1.4632 |
| 1.5531 | 2.3708 | 58750 | 1.4558 |
| 1.5329 | 2.4213 | 60000 | 1.4507 |
| 1.5481 | 2.4717 | 61250 | 1.4472 |
| 1.5336 | 2.5222 | 62500 | 1.4431 |
| 1.526 | 2.5726 | 63750 | 1.4405 |
| 1.518 | 2.6231 | 65000 | 1.4345 |
| 1.5135 | 2.6735 | 66250 | 1.4264 |
| 1.4987 | 2.7239 | 67500 | 1.4226 |
| 1.5007 | 2.7744 | 68750 | 1.4176 |
| 1.4921 | 2.8248 | 70000 | 1.4179 |
| 1.5031 | 2.8753 | 71250 | 1.4146 |
| 1.4848 | 2.9257 | 72500 | 1.4098 |
| 1.4702 | 2.9762 | 73750 | 1.4023 |
| 1.4861 | 3.0266 | 75000 | 1.4010 |
| 1.487 | 3.0770 | 76250 | 1.3963 |
| 1.4736 | 3.1275 | 77500 | 1.3923 |
| 1.4751 | 3.1779 | 78750 | 1.3879 |
| 1.4783 | 3.2284 | 80000 | 1.3858 |
| 1.4843 | 3.2788 | 81250 | 1.3795 |
| 1.4722 | 3.3293 | 82500 | 1.3771 |
| 1.4551 | 3.3797 | 83750 | 1.3754 |
| 1.4539 | 3.4302 | 85000 | 1.3729 |
| 1.4723 | 3.4806 | 86250 | 1.3646 |
| 1.4493 | 3.5310 | 87500 | 1.3658 |
| 1.4455 | 3.5815 | 88750 | 1.3610 |
| 1.4442 | 3.6319 | 90000 | 1.3573 |
| 1.4457 | 3.6824 | 91250 | 1.3540 |
| 1.4259 | 3.7328 | 92500 | 1.3534 |
| 1.4355 | 3.7833 | 93750 | 1.3470 |
| 1.4184 | 3.8337 | 95000 | 1.3435 |
| 1.4437 | 3.8841 | 96250 | 1.3416 |
| 1.4255 | 3.9346 | 97500 | 1.3377 |
| 1.4115 | 3.9850 | 98750 | 1.3358 |
| 1.4196 | 4.0355 | 100000 | 1.3351 |
| 1.4159 | 4.0859 | 101250 | 1.3292 |
| 1.4227 | 4.1364 | 102500 | 1.3302 |
| 1.4122 | 4.1868 | 103750 | 1.3270 |
| 1.3996 | 4.2372 | 105000 | 1.3207 |
| 1.4041 | 4.2877 | 106250 | 1.3210 |
| 1.3956 | 4.3381 | 107500 | 1.3187 |
| 1.392 | 4.3886 | 108750 | 1.3170 |
| 1.3943 | 4.4390 | 110000 | 1.3125 |
| 1.4143 | 4.4895 | 111250 | 1.3095 |
| 1.3939 | 4.5399 | 112500 | 1.3063 |
| 1.3802 | 4.5903 | 113750 | 1.3067 |
| 1.3908 | 4.6408 | 115000 | 1.3020 |
| 1.3841 | 4.6912 | 116250 | 1.3025 |
| 1.3821 | 4.7417 | 117500 | 1.3007 |
| 1.3774 | 4.7921 | 118750 | 1.2989 |
| 1.3807 | 4.8426 | 120000 | 1.2907 |
| 1.3643 | 4.8930 | 121250 | 1.2946 |
| 1.3704 | 4.9435 | 122500 | 1.2920 |
| 1.3685 | 4.9939 | 123750 | 1.2868 |
| 1.3794 | 5.0443 | 125000 | 1.2812 |
| 1.3646 | 5.0948 | 126250 | 1.2809 |
| 1.356 | 5.1452 | 127500 | 1.2803 |
| 1.3696 | 5.1957 | 128750 | 1.2784 |
| 1.3544 | 5.2461 | 130000 | 1.2741 |
| 1.3618 | 5.2966 | 131250 | 1.2736 |
| 1.3471 | 5.3470 | 132500 | 1.2695 |
| 1.3444 | 5.3974 | 133750 | 1.2648 |
| 1.3524 | 5.4479 | 135000 | 1.2658 |
| 1.354 | 5.4983 | 136250 | 1.2643 |
| 1.3438 | 5.5488 | 137500 | 1.2639 |
| 1.357 | 5.5992 | 138750 | 1.2599 |
| 1.3473 | 5.6497 | 140000 | 1.2617 |
| 1.3309 | 5.7001 | 141250 | 1.2568 |
| 1.3328 | 5.7505 | 142500 | 1.2511 |
| 1.3236 | 5.8010 | 143750 | 1.2511 |
| 1.3276 | 5.8514 | 145000 | 1.2507 |
| 1.3288 | 5.9019 | 146250 | 1.2466 |
| 1.3238 | 5.9523 | 147500 | 1.2456 |
| 1.3327 | 6.0028 | 148750 | 1.2484 |
| 1.3329 | 6.0532 | 150000 | 1.2424 |
| 1.3328 | 6.1037 | 151250 | 1.2361 |
| 1.307 | 6.1541 | 152500 | 1.2407 |
| 1.3285 | 6.2045 | 153750 | 1.2374 |
| 1.3097 | 6.2550 | 155000 | 1.2339 |
| 1.3115 | 6.3054 | 156250 | 1.2354 |
| 1.304 | 6.3559 | 157500 | 1.2294 |
| 1.3132 | 6.4063 | 158750 | 1.2290 |
| 1.303 | 6.4568 | 160000 | 1.2276 |
| 1.3029 | 6.5072 | 161250 | 1.2270 |
| 1.3048 | 6.5576 | 162500 | 1.2229 |
| 1.3085 | 6.6081 | 163750 | 1.2226 |
| 1.2887 | 6.6585 | 165000 | 1.2209 |
| 1.3055 | 6.7090 | 166250 | 1.2206 |
| 1.2902 | 6.7594 | 167500 | 1.2178 |
| 1.2892 | 6.8099 | 168750 | 1.2149 |
| 1.3049 | 6.8603 | 170000 | 1.2125 |
| 1.2935 | 6.9107 | 171250 | 1.2115 |
| 1.2888 | 6.9612 | 172500 | 1.2091 |
| 1.2856 | 7.0116 | 173750 | 1.2082 |
| 1.2762 | 7.0621 | 175000 | 1.2085 |
| 1.2883 | 7.1125 | 176250 | 1.2055 |
| 1.2906 | 7.1630 | 177500 | 1.2019 |
| 1.2831 | 7.2134 | 178750 | 1.2047 |
| 1.2654 | 7.2638 | 180000 | 1.1995 |
| 1.2759 | 7.3143 | 181250 | 1.1994 |
| 1.276 | 7.3647 | 182500 | 1.1992 |
| 1.2692 | 7.4152 | 183750 | 1.1974 |
| 1.2791 | 7.4656 | 185000 | 1.1940 |
| 1.2697 | 7.5161 | 186250 | 1.1930 |
| 1.2635 | 7.5665 | 187500 | 1.1889 |
| 1.2656 | 7.6170 | 188750 | 1.1926 |
| 1.2615 | 7.6675 | 190000 | 1.1828 |
| 1.2641 | 7.7179 | 191250 | 1.1852 |
| 1.2578 | 7.7684 | 192500 | 1.1791 |
| 1.2647 | 7.8188 | 193750 | 1.1782 |
| 1.2644 | 7.8692 | 195000 | 1.1777 |
| 1.2638 | 7.9197 | 196250 | 1.1752 |
| 1.2528 | 7.9701 | 197500 | 1.1748 |
| 1.2554 | 8.0206 | 198750 | 1.1746 |
| 1.2548 | 8.0710 | 200000 | 1.1726 |
| 1.2546 | 8.1215 | 201250 | 1.1698 |
| 1.247 | 8.1719 | 202500 | 1.1689 |
| 1.2478 | 8.2223 | 203750 | 1.1698 |
| 1.2578 | 8.2728 | 205000 | 1.1650 |
| 1.2527 | 8.3232 | 206250 | 1.1650 |
| 1.2612 | 8.3737 | 207500 | 1.1639 |
| 1.2339 | 8.4241 | 208750 | 1.1635 |
| 1.2422 | 8.4746 | 210000 | 1.1633 |
| 1.2311 | 8.5250 | 211250 | 1.1617 |
| 1.2552 | 8.5754 | 212500 | 1.1585 |
| 1.2383 | 8.6259 | 213750 | 1.1561 |
| 1.2406 | 8.6763 | 215000 | 1.1555 |
| 1.2329 | 8.7268 | 216250 | 1.1551 |
| 1.2392 | 8.7772 | 217500 | 1.1552 |
| 1.2301 | 8.8277 | 218750 | 1.1536 |
| 1.2262 | 8.8781 | 220000 | 1.1483 |
| 1.2284 | 8.9286 | 221250 | 1.1509 |
| 1.2259 | 8.9790 | 222500 | 1.1529 |
| 1.2204 | 9.0294 | 223750 | 1.1474 |
| 1.237 | 9.0799 | 225000 | 1.1471 |
| 1.2432 | 9.1303 | 226250 | 1.1439 |
| 1.2145 | 9.1808 | 227500 | 1.1473 |
| 1.2132 | 9.2312 | 228750 | 1.1428 |
| 1.2178 | 9.2817 | 230000 | 1.1426 |
| 1.2138 | 9.3321 | 231250 | 1.1416 |
| 1.2204 | 9.3825 | 232500 | 1.1422 |
| 1.2233 | 9.4330 | 233750 | 1.1402 |
| 1.2048 | 9.4834 | 235000 | 1.1370 |
| 1.2203 | 9.5339 | 236250 | 1.1389 |
| 1.2156 | 9.5843 | 237500 | 1.1375 |
| 1.2131 | 9.6348 | 238750 | 1.1367 |
| 1.2215 | 9.6852 | 240000 | 1.1387 |
| 1.2152 | 9.7356 | 241250 | 1.1347 |
| 1.2179 | 9.7861 | 242500 | 1.1321 |
| 1.2166 | 9.8365 | 243750 | 1.1359 |
| 1.2171 | 9.8870 | 245000 | 1.1343 |
| 1.208 | 9.9374 | 246250 | 1.1321 |
| 1.2105 | 9.9879 | 247500 | 1.1332 |
Framework versions
- Transformers 4.41.2
- Pytorch 2.3.1+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1