Hugging Face's logo

rccmsu
/
ruadapt_llama2_7b_v0.1

Text Generation
Transformers
PyTorch
Russian
llama
text-generation-inference
Inference Endpoints
Model card Files Community
1
ruadapt_llama2_7b_v0.1 / README.md
rccmsu's picture
rccmsu
Update README.md
7b62092
preview code
|
raw
history blame
18.9 kB
metadata
license: llama2
language:
  - ru
metrics:
  - accuracy

ruadapt_llama2_7b_v0.1

This model is a fine-tuned (embeddings, lm head) version of TheBloke/Llama-2-7B-fp16 on the Russian dataset (33GB). It achieves the following results on the evaluation set:

  • Loss: 2.7569
  • Accuracy: 0.4617

Model description

Russian adaptation of LLaMa-2-7B by replacing the tokenizer. Paper: Tikhomirov M.M., Chernyshev D.I., Impact of Tokenization on LLaMa Russian Adaptation (will be soon)

Intended uses & limitations

LLAMA 2 COMMUNITY LICENSE AGREEMENT

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 6
  • eval_batch_size: 6
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 16
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 192
  • total_eval_batch_size: 96
  • optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
  • lr_scheduler_type: linear
  • num_epochs: 2.0

Training results

Training Loss Epoch Step Validation Loss Accuracy
4.9167 0.01 1000 4.8647 0.2686
3.9697 0.01 2000 3.9705 0.3409
3.6398 0.02 3000 3.6476 0.3694
3.468 0.03 4000 3.4784 0.3850
3.3567 0.04 5000 3.3733 0.3953
3.2828 0.04 6000 3.2999 0.4026
3.2235 0.05 7000 3.2453 0.4081
3.1898 0.06 8000 3.2028 0.4125
3.1552 0.07 9000 3.1683 0.4160
3.1068 0.07 10000 3.1397 0.4190
3.1019 0.08 11000 3.1152 0.4217
3.0849 0.09 12000 3.0942 0.4239
3.0561 0.09 13000 3.0761 0.4256
3.0429 0.1 14000 3.0595 0.4277
3.035 0.11 15000 3.0451 0.4293
3.0077 0.12 16000 3.0322 0.4306
3.0008 0.12 17000 3.0200 0.4320
2.9952 0.13 18000 3.0093 0.4330
2.9825 0.14 19000 2.9996 0.4341
2.9781 0.14 20000 2.9903 0.4351
2.957 0.15 21000 2.9821 0.4360
2.9676 0.16 22000 2.9738 0.4368
2.9513 0.17 23000 2.9663 0.4376
2.9475 0.17 24000 2.9594 0.4385
2.9406 0.18 25000 2.9531 0.4391
2.9387 0.19 26000 2.9473 0.4398
2.9353 0.2 27000 2.9416 0.4403
2.9208 0.2 28000 2.9363 0.4411
2.9142 0.21 29000 2.9310 0.4415
2.9167 0.22 30000 2.9265 0.4419
2.9069 0.22 31000 2.9214 0.4425
2.9067 0.23 32000 2.9168 0.4430
2.8978 0.24 33000 2.9128 0.4434
2.8982 0.25 34000 2.9088 0.4438
2.8856 0.25 35000 2.9050 0.4444
2.8981 0.26 36000 2.9013 0.4445
2.8813 0.27 37000 2.8977 0.4450
2.8765 0.27 38000 2.8944 0.4453
2.879 0.28 39000 2.8910 0.4458
2.8738 0.29 40000 2.8878 0.4462
2.8671 0.3 41000 2.8851 0.4465
2.866 0.3 42000 2.8820 0.4468
2.8561 0.31 43000 2.8791 0.4473
2.8601 0.32 44000 2.8765 0.4477
2.8518 0.33 45000 2.8741 0.4479
2.8577 0.33 46000 2.8713 0.4483
2.8588 0.34 47000 2.8691 0.4484
2.8584 0.35 48000 2.8666 0.4487
2.8527 0.35 49000 2.8646 0.4488
2.8425 0.36 50000 2.8624 0.4490
2.8457 0.37 51000 2.8601 0.4494
2.849 0.38 52000 2.8580 0.4496
2.8431 0.38 53000 2.8560 0.4499
2.8463 0.39 54000 2.8540 0.4501
2.8437 0.4 55000 2.8521 0.4504
2.845 0.41 56000 2.8505 0.4505
2.8218 0.41 57000 2.8486 0.4508
2.8366 0.42 58000 2.8470 0.4509
2.8339 0.43 59000 2.8453 0.4512
2.8338 0.43 60000 2.8437 0.4511
2.8237 0.44 61000 2.8420 0.4513
2.8334 0.45 62000 2.8405 0.4515
2.8229 0.46 63000 2.8388 0.4518
2.8214 0.46 64000 2.8373 0.4519
2.8245 0.47 65000 2.8356 0.4522
2.822 0.48 66000 2.8343 0.4524
2.8139 0.48 67000 2.8331 0.4526
2.8201 0.49 68000 2.8317 0.4526
2.8132 0.5 69000 2.8305 0.4527
2.8138 0.51 70000 2.8290 0.4530
2.8171 0.51 71000 2.8279 0.4530
2.8123 0.52 72000 2.8267 0.4532
2.8118 0.53 73000 2.8255 0.4534
2.8183 0.54 74000 2.8243 0.4536
2.8052 0.54 75000 2.8233 0.4536
2.8101 0.55 76000 2.8220 0.4538
2.8021 0.56 77000 2.8209 0.4540
2.8076 0.56 78000 2.8196 0.4540
2.7937 0.57 79000 2.8190 0.4542
2.8057 0.58 80000 2.8179 0.4541
2.8082 0.59 81000 2.8168 0.4545
2.7986 0.59 82000 2.8157 0.4546
2.8062 0.6 83000 2.8150 0.4545
2.7981 0.61 84000 2.8138 0.4546
2.8041 0.61 85000 2.8130 0.4546
2.7978 0.62 86000 2.8118 0.4549
2.8016 0.63 87000 2.8109 0.4549
2.7901 0.64 88000 2.8099 0.4551
2.8075 0.64 89000 2.8093 0.4553
2.7915 0.65 90000 2.8084 0.4552
2.7916 0.66 91000 2.8074 0.4555
2.7751 0.67 92000 2.8068 0.4554
2.7896 0.67 93000 2.8059 0.4556
2.7886 0.68 94000 2.8051 0.4557
2.7909 0.69 95000 2.8044 0.4557
2.7926 0.69 96000 2.8035 0.4558
2.7931 0.7 97000 2.8028 0.4560
2.7838 0.71 98000 2.8020 0.4562
2.779 0.72 99000 2.8014 0.4561
2.7922 0.72 100000 2.8006 0.4562
2.7786 0.73 101000 2.7999 0.4562
2.7791 0.74 102000 2.7992 0.4563
2.7908 0.74 103000 2.7984 0.4565
2.7872 0.75 104000 2.7978 0.4566
2.7763 0.76 105000 2.7972 0.4567
2.7785 0.77 106000 2.7966 0.4568
2.7861 0.77 107000 2.7960 0.4568
2.784 0.78 108000 2.7953 0.4570
2.7804 0.79 109000 2.7944 0.4571
2.7828 0.8 110000 2.7940 0.4570
2.7761 0.8 111000 2.7933 0.4571
2.7797 0.81 112000 2.7928 0.4571
2.7792 0.82 113000 2.7922 0.4573
2.7819 0.82 114000 2.7915 0.4573
2.7837 0.83 115000 2.7910 0.4573
2.781 0.84 116000 2.7906 0.4575
2.7765 0.85 117000 2.7898 0.4577
2.7778 0.85 118000 2.7895 0.4575
2.776 0.86 119000 2.7887 0.4577
2.7719 0.87 120000 2.7883 0.4578
2.7759 0.88 121000 2.7878 0.4579
2.7654 0.88 122000 2.7874 0.4578
2.7661 0.89 123000 2.7868 0.4580
2.7718 0.9 124000 2.7861 0.4580
2.7775 0.9 125000 2.7858 0.4580
2.7835 0.91 126000 2.7855 0.4580
2.768 0.92 127000 2.7848 0.4581
2.7701 0.93 128000 2.7843 0.4582
2.7682 0.93 129000 2.7838 0.4583
2.7595 0.94 130000 2.7834 0.4583
2.7627 0.95 131000 2.7831 0.4583
2.7716 0.95 132000 2.7827 0.4584
2.7719 0.96 133000 2.7821 0.4585
2.7723 0.97 134000 2.7816 0.4583
2.7736 0.98 135000 2.7812 0.4585
2.7646 0.98 136000 2.7809 0.4586
2.76 0.99 137000 2.7805 0.4586
2.7659 1.0 138000 2.7803 0.4586
2.7604 1.01 139000 2.7799 0.4587
2.7597 1.01 140000 2.7794 0.4587
2.7551 1.02 141000 2.7791 0.4588
2.7619 1.03 142000 2.7788 0.4588
2.7658 1.03 143000 2.7785 0.4589
2.751 1.04 144000 2.7781 0.4589
2.7589 1.05 145000 2.7778 0.4590
2.7459 1.06 146000 2.7776 0.4590
2.7646 1.06 147000 2.7771 0.4591
2.7529 1.07 148000 2.7768 0.4589
2.7573 1.08 149000 2.7764 0.4592
2.754 1.08 150000 2.7762 0.4591
2.7553 1.09 151000 2.7759 0.4591
2.7485 1.1 152000 2.7755 0.4593
2.7558 1.11 153000 2.7752 0.4593
2.7563 1.11 154000 2.7748 0.4593
2.7557 1.12 155000 2.7747 0.4594
2.7593 1.13 156000 2.7744 0.4592
2.752 1.14 157000 2.7741 0.4593
2.748 1.14 158000 2.7737 0.4593
2.7549 1.15 159000 2.7735 0.4594
2.7455 1.16 160000 2.7733 0.4596
2.7582 1.16 161000 2.7731 0.4594
2.7532 1.17 162000 2.7728 0.4595
2.7496 1.18 163000 2.7724 0.4595
2.75 1.19 164000 2.7721 0.4596
2.7517 1.19 165000 2.7718 0.4597
2.7522 1.2 166000 2.7716 0.4597
2.7514 1.21 167000 2.7713 0.4599
2.7515 1.22 168000 2.7711 0.4598
2.7493 1.22 169000 2.7708 0.4598
2.7491 1.23 170000 2.7705 0.4598
2.7552 1.24 171000 2.7704 0.4599
2.7536 1.24 172000 2.7700 0.4600
2.7485 1.25 173000 2.7697 0.4599
2.7455 1.26 174000 2.7697 0.4599
2.7516 1.27 175000 2.7694 0.4599
2.754 1.27 176000 2.7690 0.4600
2.7489 1.28 177000 2.7690 0.4598
2.7491 1.29 178000 2.7686 0.4601
2.7432 1.29 179000 2.7684 0.4600
2.7388 1.3 180000 2.7681 0.4602
2.7501 1.31 181000 2.7679 0.4602
2.7526 1.32 182000 2.7675 0.4603
2.7478 1.32 183000 2.7674 0.4603
2.7491 1.33 184000 2.7670 0.4604
2.7505 1.34 185000 2.7670 0.4604
2.7436 1.35 186000 2.7666 0.4605
2.7389 1.35 187000 2.7665 0.4603
2.7564 1.36 188000 2.7662 0.4604
2.7464 1.37 189000 2.7661 0.4604
2.7459 1.37 190000 2.7659 0.4605
2.7481 1.38 191000 2.7657 0.4605
2.7458 1.39 192000 2.7655 0.4604
2.7427 1.4 193000 2.7653 0.4605
2.741 1.4 194000 2.7651 0.4606
2.7488 1.41 195000 2.7649 0.4606
2.7353 1.42 196000 2.7647 0.4605
2.7503 1.42 197000 2.7645 0.4607
2.7446 1.43 198000 2.7644 0.4607
2.748 1.44 199000 2.7642 0.4607
2.7394 1.45 200000 2.7641 0.4607
2.7403 1.45 201000 2.7638 0.4607
2.7467 1.46 202000 2.7637 0.4607
2.7532 1.47 203000 2.7635 0.4608
2.7431 1.48 204000 2.7634 0.4609
2.7433 1.48 205000 2.7632 0.4608
2.7436 1.49 206000 2.7630 0.4609
2.747 1.5 207000 2.7628 0.4609
2.7395 1.5 208000 2.7626 0.4609
2.7443 1.51 209000 2.7624 0.4609
2.7395 1.52 210000 2.7623 0.4608
2.7353 1.53 211000 2.7621 0.4608
2.7401 1.53 212000 2.7618 0.4610
2.7371 1.54 213000 2.7617 0.4610
2.7458 1.55 214000 2.7616 0.4610
2.7416 1.56 215000 2.7615 0.4611
2.7434 1.56 216000 2.7614 0.4611
2.7456 1.57 217000 2.7614 0.4611
2.7499 1.58 218000 2.7611 0.4611
2.744 1.58 219000 2.7609 0.4611
2.7375 1.59 220000 2.7608 0.4611
2.7428 1.6 221000 2.7606 0.4611
2.7442 1.61 222000 2.7606 0.4611
2.7395 1.61 223000 2.7604 0.4612
2.7445 1.62 224000 2.7602 0.4612
2.7394 1.63 225000 2.7602 0.4611
2.7403 1.63 226000 2.7599 0.4612
2.738 1.64 227000 2.7599 0.4612
2.7332 1.65 228000 2.7597 0.4613
2.7388 1.66 229000 2.7596 0.4613
2.743 1.66 230000 2.7595 0.4613
2.7368 1.67 231000 2.7593 0.4613
2.7426 1.68 232000 2.7592 0.4614
2.7332 1.69 233000 2.7591 0.4614
2.7413 1.69 234000 2.7590 0.4614
2.735 1.7 235000 2.7589 0.4613
2.7393 1.71 236000 2.7589 0.4614
2.7382 1.71 237000 2.7587 0.4615
2.7403 1.72 238000 2.7587 0.4615
2.7436 1.73 239000 2.7586 0.4615
2.7422 1.74 240000 2.7585 0.4615
2.7257 1.74 241000 2.7584 0.4614
2.7351 1.75 242000 2.7583 0.4615
2.7391 1.76 243000 2.7582 0.4615
2.7495 1.76 244000 2.7581 0.4615
2.7399 1.77 245000 2.7580 0.4614
2.7435 1.78 246000 2.7580 0.4616
2.7414 1.79 247000 2.7579 0.4615
2.7478 1.79 248000 2.7578 0.4616
2.7299 1.8 249000 2.7577 0.4616
2.7401 1.81 250000 2.7576 0.4616
2.7395 1.82 251000 2.7575 0.4616
2.7399 1.82 252000 2.7574 0.4616
2.7413 1.83 253000 2.7574 0.4616
2.7294 1.84 254000 2.7573 0.4616
2.7329 1.84 255000 2.7572 0.4616
2.7454 1.85 256000 2.7572 0.4617
2.7343 1.86 257000 2.7571 0.4617
2.7356 1.87 258000 2.7571 0.4617
2.7462 1.87 259000 2.7570 0.4617
2.7375 1.88 260000 2.7569 0.4617
2.7368 1.89 261000 2.7569 0.4618
2.7452 1.89 262000 2.7569 0.4617
2.7394 1.9 263000 2.7568 0.4617
2.7378 1.91 264000 2.7568 0.4618
2.7446 1.92 265000 2.7567 0.4618
2.7436 1.92 266000 2.7567 0.4618
2.7505 1.93 267000 2.7567 0.4618
2.7493 1.94 268000 2.7566 0.4618
2.7391 1.95 269000 2.7566 0.4618
2.7431 1.95 270000 2.7566 0.4617
2.7387 1.96 271000 2.7565 0.4618
2.741 1.97 272000 2.7565 0.4618
2.7343 1.97 273000 2.7565 0.4618
2.7378 1.98 274000 2.7564 0.4618
2.737 1.99 275000 2.7564 0.4618
2.7397 2.0 276000 2.7564 0.4618

Framework versions

  • Transformers 4.34.0
  • Pytorch 2.0.1+cu118
  • Datasets 2.14.5
  • Tokenizers 0.14.1