End of training

README.md

@@ -0,0 +1,101 @@
+---
+base_model: roneneldan/TinyStories-33M
+library_name: Distily
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_TinyStories-33M
+  results: []
+---
+
+# distily_TinyStories-33M
+
+This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 5885.9341
+- eval_frwikippl: 24294.9414
+- eval_zhwikippl: 264331.3438
+- eval_loss: 0.3987
+- eval_runtime: 51.5838
+- eval_samples_per_second: 48.465
+- eval_steps_per_second: 6.068
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- train_embeddings: True
+- learning_rate: 4e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 8.1416 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 20633.1680 | 131577.2812 |  |  |  |  | 7615.4468 |
+| 0 | 0 | 55266.375 | 57180.4375 | 6.2843 | 26.4237 | 94.612 | 11.845 | 56806.5430 |
+| 1000 | 0.0323 | 11414.3389 | 87921.1172 | 0.7142 | 26.3405 | 94.911 | 11.883 | 611931.1875 |
+| 2000 | 0.0646 | 8814.8682 | 53295.2305 | 0.6287 | 51.0412 | 48.98 | 6.132 | 507315.5625 |
+| 3000 | 0.0970 | 8020.6040 | 41652.3320 | 0.5662 | 29.4187 | 84.98 | 10.639 | 268242.625 |
+| 4000 | 0.1293 | 7153.7090 | 33178.5977 | 0.5197 | 40.0478 | 62.425 | 7.816 | 315367.9062 |
+| 5000 | 0.1616 | 6865.2617 | 31042.1875 | 0.4833 | 36.655 | 68.203 | 8.539 | 372857.25 |
+| 6000 | 0.1939 | 6828.5781 | 30924.2324 | 0.4539 | 47.1811 | 52.987 | 6.634 | 379690.5 |
+| 7000 | 0.2263 | 6329.1855 | 28375.3984 | 0.4331 | 51.6027 | 48.447 | 6.066 | 325812.875 |
+| 8000 | 0.2586 | 6229.7119 | 28592.2773 | 0.4123 | 51.6184 | 48.432 | 6.064 | 318159.5 |
+| 9000 | 0.2909 | 5885.9341 | 24294.9414 | 0.3987 | 51.5838 | 48.465 | 6.068 | 264331.3438 |
+| 10000 | 0.3232 | 5634.5898 | 24401.3828 | 0.3856 | 51.6233 | 48.428 | 6.063 | 248118.4062 |
+| 11000 | 0.3555 | 5849.9346 | 26113.8555 | 0.3761 | 51.5949 | 48.454 | 6.066 | 255583.9844 |
+| 12000 | 0.3879 | 5588.8325 | 23138.0430 | 0.3666 | 51.5384 | 48.508 | 6.073 | 255106.6875 |
+| 13000 | 0.4202 | 5498.4355 | 23102.1699 | 0.3618 | 51.6778 | 48.377 | 6.057 | 244239.3125 |
+| 14000 | 0.4525 | 5495.8716 | 24775.8398 | 0.3530 | 51.4537 | 48.587 | 6.083 | 271776.25 |
+| 15000 | 0.4848 | 5449.1309 | 23173.9512 | 0.3490 | 51.6347 | 48.417 | 6.062 | 235716.0625 |
+| 16000 | 0.5172 | 5464.8057 | 25348.3184 | 0.3430 | 48.3546 | 51.701 | 6.473 | 305992.3125 |
+| 17000 | 0.5495 | 5289.8618 | 23652.6602 | 0.3426 | 45.4673 | 54.985 | 6.884 | 290930.0625 |
+| 18000 | 0.5818 | 5362.6548 | 23393.9375 | 0.3378 | 42.8681 | 58.318 | 7.301 | 237739.0938 |
+| 19000 | 0.6141 | 5970.6357 | 32165.1016 | 0.3332 | 38.4757 | 64.976 | 8.135 | 492760.0312 |
+| 20000 | 0.6465 | 5680.7217 | 30225.7988 | 0.3322 | 31.9943 | 78.139 | 9.783 | 391742.4062 |
+| 21000 | 0.6788 | 5494.1685 | 27750.1914 | 0.3288 | 49.7191 | 50.283 | 6.295 | 288762.6875 |
+| 22000 | 0.7111 | 5693.0815 | 24919.4883 | 0.3272 | 49.6244 | 50.378 | 6.307 | 263274.4375 |
+| 23000 | 0.7434 | 5303.4346 | 25441.4375 | 0.3230 | 50.6137 | 49.394 | 6.184 | 261801.9844 |
+| 24000 | 0.7757 | 5458.4463 | 26499.6543 | 0.3217 | 51.4227 | 48.617 | 6.087 | 229626.5781 |
+| 25000 | 0.8081 | 5728.1162 | 28263.5859 | 0.3203 | 51.6717 | 48.382 | 6.057 | 258605.3594 |
+| 26000 | 0.8404 | 5226.1689 | 23493.1152 | 0.3186 | 51.4811 | 48.562 | 6.08 | 180660.6719 |
+| 27000 | 0.8727 | 5192.1890 | 22039.3262 | 0.3165 | 51.6376 | 48.414 | 6.061 | 194013.875 |
+| 28000 | 0.9050 | 5418.7476 | 22450.2344 | 0.3169 | 51.6539 | 48.399 | 6.06 | 182503.5312 |
+| 29000 | 0.9374 | 5170.8613 | 23860.3691 | 0.3141 | 51.4944 | 48.549 | 6.078 | 197516.9531 |
+| 30000 | 0.9697 | 5569.3379 | 25081.6641 | 0.3130 | 51.3337 | 48.701 | 6.097 | 160202.3281 |
+| 30938 | 1.0 | 5306.7280 | 25078.125 | 0.3130 | 51.5266 | 48.519 | 6.075 | 179410.5625 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.21.0

config.json

@@ -0,0 +1,47 @@
+{
+  "_name_or_path": "roneneldan/TinyStories-33M",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPTNeoForCausalLM"
+  ],
+  "attention_dropout": 0,
+  "attention_layers": [
+    "global",
+    "local",
+    "global",
+    "local"
+  ],
+  "attention_types": [
+    [
+      [
+        "global",
+        "local"
+      ],
+      2
+    ]
+  ],
+  "bos_token_id": 50256,
+  "classifier_dropout": 0.1,
+  "embed_dropout": 0,
+  "eos_token_id": 50256,
+  "gradient_checkpointing": false,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": null,
+  "layer_norm_epsilon": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "gpt_neo",
+  "num_heads": 16,
+  "num_layers": 4,
+  "resid_dropout": 0,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.44.0",
+  "use_cache": true,
+  "vocab_size": 50257,
+  "window_size": 256
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46d62c1acd835b0d3302fb89efc4fb14d8a4a8ae6c18fac57667cb92d59710fe
+size 137033984

runs/Aug15_10-33-04_77e473d64567/events.out.tfevents.1723718487.77e473d64567

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9e8ed2bb568428b38afd99b528bcd9410c16b42a623ef4d3b9b35fa81630bc4
+size 1015978

runs/Aug15_10-54-56_77e473d64567/events.out.tfevents.1723719411.77e473d64567

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6256177912decab7694178db0c27e75c0578be219180c423a774147db1bf1b26
+size 8430941

runs/Aug15_10-54-56_77e473d64567/events.out.tfevents.1723730393.77e473d64567

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71f201c7055aae902cf28133ef9a872189f38c89c0f5aa26ddd1ece58d0ec828
+size 253

runs/Aug15_11-00-37_77e473d64567/events.out.tfevents.1723719744.77e473d64567

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dce729974b10a280f181af8ace309458ec25530b44cbf52615ae3a037bcc148
+size 310567

runs/Aug15_11-10-45_77e473d64567/events.out.tfevents.1723720351.77e473d64567

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c09f845e0d0c52be5ecb21a44599bb766deb6d857aceb85434bb87d49d8589a
+size 7347370

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47d34912321227943c575b1e98b78c24af82a612e06d98c18bbed312010075ac
+size 907106564