Hugging Face's logo

lapp0
/
distily_bench_gpt2_simple_objectives2

TensorBoard
Safetensors
Distily
gpt2
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Metrics Community
lapp0 commited on
Commit
13f66f0
•
1 Parent(s): e3f2787

End of training

Browse files
Files changed (2) hide show
  1. README.md +34 -34
  2. logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_cosine_distance_loss())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_los/events.out.tfevents.1723556313.93d6cbb3ad53 +3 -0
README.md CHANGED
@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
- - eval_enwikippl: 50740220.0
20
- - eval_frwikippl: 39742000.0
21
- - eval_zhwikippl: 41711904.0
22
- - eval_loss: -1133756.375
23
- - eval_runtime: 32.9889
24
- - eval_samples_per_second: 60.627
25
- - eval_steps_per_second: 7.578
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
- - distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:cross_entropy()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
@@ -62,32 +62,32 @@ Peak GPU Memory: 10.3934 GB
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
- | 0 | 0 | 54069.2930 | 57285.3438 | -552648.6875 | 32.884 | 60.82 | 7.602 | 54227.1016 |
66
- | 1000 | 0.0404 | 58406536.0 | 55278164.0 | -1128988.625 | 32.9702 | 60.661 | 7.583 | 60044060.0 |
67
- | 2000 | 0.0808 | 47996096.0 | 32493628.0 | -1115226.125 | 32.92 | 60.753 | 7.594 | 30615724.0 |
68
- | 3000 | 0.1212 | 53309036.0 | 52586468.0 | -1142620.125 | 32.8831 | 60.822 | 7.603 | 63406108.0 |
69
- | 4000 | 0.1616 | 48853388.0 | 39451680.0 | -1129824.25 | 32.9886 | 60.627 | 7.578 | 40786548.0 |
70
- | 5000 | 0.2020 | 48792764.0 | 39496180.0 | -1129889.75 | 32.94 | 60.716 | 7.59 | 41224608.0 |
71
- | 6000 | 0.2424 | 49218992.0 | 39786900.0 | -1130381.25 | 32.8582 | 60.868 | 7.608 | 41379088.0 |
72
- | 7000 | 0.2828 | 48641396.0 | 39540800.0 | -1130201.125 | 33.0051 | 60.597 | 7.575 | 41202596.0 |
73
- | 8000 | 0.3232 | 48641396.0 | 39742000.0 | -1130168.375 | 33.0186 | 60.572 | 7.572 | 41312844.0 |
74
- | 9000 | 0.3636 | 50740220.0 | 39742000.0 | -1133756.375 | 32.9889 | 60.627 | 7.578 | 41711904.0 |
75
- | 10000 | 0.4040 | 50945516.0 | 40215528.0 | -1134297.125 | 33.0575 | 60.501 | 7.563 | 42227584.0 |
76
- | 11000 | 0.4444 | 51968464.0 | 40443060.0 | -1135476.75 | 33.1355 | 60.358 | 7.545 | 42589960.0 |
77
- | 12000 | 0.4848 | 50410312.0 | 40465824.0 | -1131806.75 | 32.913 | 60.766 | 7.596 | 42453708.0 |
78
- | 13000 | 0.5253 | 49020652.0 | 39674820.0 | -1130709.0 | 33.1318 | 60.365 | 7.546 | 41778864.0 |
79
- | 14000 | 0.5657 | 47536192.0 | 38998080.0 | -1128382.5 | 33.163 | 60.308 | 7.539 | 40504356.0 |
80
- | 15000 | 0.6061 | 47936448.0 | 38822560.0 | -1127694.375 | 33.0155 | 60.578 | 7.572 | 40461116.0 |
81
- | 16000 | 0.6465 | 49035896.0 | 42119312.0 | -1133248.5 | 33.0001 | 60.606 | 7.576 | 44687476.0 |
82
- | 17000 | 0.6869 | 49081556.0 | 43840280.0 | -1134542.875 | 33.0658 | 60.485 | 7.561 | 47391796.0 |
83
- | 18000 | 0.7273 | 48823112.0 | 45311016.0 | -1135951.875 | 32.8757 | 60.835 | 7.604 | 48054540.0 |
84
- | 19000 | 0.7677 | 51358604.0 | 46384208.0 | -1138917.375 | 33.0459 | 60.522 | 7.565 | 52342104.0 |
85
- | 20000 | 0.8081 | 52308568.0 | 47335696.0 | -1139867.625 | 33.1167 | 60.393 | 7.549 | 52819580.0 |
86
- | 21000 | 0.8485 | 49633508.0 | 44375164.0 | -1133396.0 | 33.0584 | 60.499 | 7.562 | 45774616.0 |
87
- | 22000 | 0.8889 | 48853388.0 | 44827992.0 | -1131577.375 | 32.9397 | 60.717 | 7.59 | 48441136.0 |
88
- | 23000 | 0.9293 | 49726084.0 | 45221656.0 | -1133002.75 | 33.0244 | 60.561 | 7.57 | 49513708.0 |
89
- | 24000 | 0.9697 | 53292568.0 | 48497804.0 | -1141473.25 | 33.1059 | 60.412 | 7.552 | 53730116.0 |
90
- | 24750 | 1.0 | 59081604.0 | 55590812.0 | -1151893.5 | 33.0542 | 60.507 | 7.563 | 59374192.0 |
91
 
92
  ### Framework versions
93
  - Distily 0.2.0
 
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
+ - eval_enwikippl: 26585.3379
20
+ - eval_frwikippl: 34195.0625
21
+ - eval_zhwikippl: 50038.4062
22
+ - eval_loss: 0.0690
23
+ - eval_runtime: 32.5862
24
+ - eval_samples_per_second: 61.376
25
+ - eval_steps_per_second: 7.672
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
 
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
+ - distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:cosine_distance_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
 
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
+ | 0 | 0 | 56797.875 | 58468.6992 | 1.0 | 32.7055 | 61.152 | 7.644 | 59002.2891 |
66
+ | 1000 | 0.0404 | 27961.6797 | 33253.5469 | 0.0687 | 32.6144 | 61.323 | 7.665 | 48954.4688 |
67
+ | 2000 | 0.0808 | 28987.5293 | 34209.5430 | 0.0692 | 32.5898 | 61.369 | 7.671 | 49453.8398 |
68
+ | 3000 | 0.1212 | 31211.6582 | 36006.2383 | 0.0684 | 32.5399 | 61.463 | 7.683 | 56097.9492 |
69
+ | 4000 | 0.1616 | 30200.7598 | 35859.2969 | 0.0683 | 32.7214 | 61.122 | 7.64 | 55147.2305 |
70
+ | 5000 | 0.2020 | 29574.1465 | 34793.3281 | 0.0687 | 32.5565 | 61.432 | 7.679 | 52868.5352 |
71
+ | 6000 | 0.2424 | 28607.3867 | 34403.0312 | 0.0681 | 32.4239 | 61.683 | 7.71 | 52629.0312 |
72
+ | 7000 | 0.2828 | 25131.9219 | 33173.9180 | 0.0687 | 32.5743 | 61.398 | 7.675 | 52278.8164 |
73
+ | 8000 | 0.3232 | 27496.5820 | 34228.8320 | 0.0682 | 32.6299 | 61.293 | 7.662 | 52896.7773 |
74
+ | 9000 | 0.3636 | 26585.3379 | 34195.0625 | 0.0690 | 32.5862 | 61.376 | 7.672 | 50038.4062 |
75
+ | 10000 | 0.4040 | 27479.4902 | 33309.8555 | 0.0692 | 32.6173 | 61.317 | 7.665 | 53123.3086 |
76
+ | 11000 | 0.4444 | 29032.5703 | 32465.8555 | 0.0687 | 32.5477 | 61.448 | 7.681 | 54270.5625 |
77
+ | 12000 | 0.4848 | 25965.1055 | 33366.2578 | 0.0679 | 32.6589 | 61.239 | 7.655 | 56639.9258 |
78
+ | 13000 | 0.5253 | 28022.5176 | 34031.5195 | 0.0687 | 32.7256 | 61.114 | 7.639 | 55873.6484 |
79
+ | 14000 | 0.5657 | 29721.4824 | 33602.3359 | 0.0682 | 32.4915 | 61.555 | 7.694 | 54241.5859 |
80
+ | 15000 | 0.6061 | 23164.0742 | 32511.6797 | 0.0678 | 32.6017 | 61.347 | 7.668 | 52727.5469 |
81
+ | 16000 | 0.6465 | 22154.2578 | 34514.7969 | 0.0686 | 32.5602 | 61.425 | 7.678 | 55412.9883 |
82
+ | 17000 | 0.6869 | 28816.9688 | 36942.2734 | 0.0689 | 32.582 | 61.384 | 7.673 | 54677.9062 |
83
+ | 18000 | 0.7273 | 30663.9199 | 36817.4492 | 0.0686 | 32.6509 | 61.254 | 7.657 | 52769.8047 |
84
+ | 19000 | 0.7677 | 31843.2832 | 37875.9023 | 0.0682 | 32.5752 | 61.396 | 7.675 | 53094.9453 |
85
+ | 20000 | 0.8081 | 26705.3535 | 35377.125 | 0.0677 | 32.5515 | 61.441 | 7.68 | 55873.6484 |
86
+ | 21000 | 0.8485 | 31013.5449 | 35662.6172 | 0.0671 | 32.8968 | 60.796 | 7.6 | 53579.2930 |
87
+ | 22000 | 0.8889 | 31917.5293 | 33950.0234 | 0.0672 | 32.7146 | 61.135 | 7.642 | 53779.9688 |
88
+ | 23000 | 0.9293 | 30907.7520 | 34783.5430 | 0.0678 | 32.8995 | 60.791 | 7.599 | 53564.9883 |
89
+ | 24000 | 0.9697 | 30893.3711 | 34383.6484 | 0.0676 | 32.7142 | 61.135 | 7.642 | 54039.1367 |
90
+ | 24750 | 1.0 | 37617.5469 | 41219.9453 | 0.0685 | 32.7176 | 61.129 | 7.641 | 52572.8477 |
91
 
92
  ### Framework versions
93
  - Distily 0.2.0
logs/distillation_objective=MultiObjective(logits_weight_1__logits_loss_fn_(fn_cosine_distance_loss())__activations_weight_0.2__activations_loss_fn_(fn_soft_mse_loss())__attentions_weight_0__attentions_los/events.out.tfevents.1723556313.93d6cbb3ad53 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fbad053fdd49cf5c00272d57c7b50fd2752f74c9fda29f258f81a570cd53ca33
3
+ size 253