---
base_model: roneneldan/TinyStories-33M
library_name: Distily
tags:
- generated_from_trainer
model-index:
- name: distily_bench_obj_cross_v2.4
  results: []
---

# distily_bench_obj_cross_v2.4

This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).

The [Distily](https://github.com/lapp0/distily) library was used for this distillation.

It achieves the following results on the evaluation set:
- eval_enwikippl: 49928.3438
- eval_frwikippl: 60082.1211
- eval_zhwikippl: 75499.0547
- eval_tinystoriesppl: 44922.0352
- eval_loss: 6.1235
- eval_runtime: 13.0526
- eval_samples_per_second: 76.613
- eval_steps_per_second: 9.577

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed
-->

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
- train_embeddings: True
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1.0

### Resource Usage
Peak GPU Memory: 8.0568 GB

### Eval-Phase Metrics
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
| 0 | 0 | 77152.8516 | 72247.2109 | 6.5107 | 12.9698 | 77.102 | 9.638 | 77917.4219 | 77892.9766 |
| 500 | 0.0404 | 49943.8203 | 60082.1211 | 6.1230 | 12.9612 | 77.153 | 9.644 | 44951.7734 | 75499.0547 |
| 1000 | 0.0808 | 49943.8203 | 60082.1211 | 6.1230 | 12.975 | 77.071 | 9.634 | 44951.7734 | 75499.0547 |
| 1500 | 0.1212 | 49943.8203 | 60082.1211 | 6.1230 | 12.9603 | 77.159 | 9.645 | 44944.3164 | 75499.0547 |
| 2000 | 0.1616 | 49943.8203 | 60082.1211 | 6.1235 | 12.9533 | 77.2 | 9.65 | 44922.0352 | 75499.0547 |
| 2500 | 0.2020 | 49943.8203 | 60082.1211 | 6.1233 | 12.9601 | 77.16 | 9.645 | 44922.0352 | 75499.0547 |
| 3000 | 0.2424 | 49920.5820 | 60082.1211 | 6.1233 | 12.9557 | 77.186 | 9.648 | 44907.2148 | 75499.0547 |
| 3500 | 0.2828 | 49920.5820 | 60082.1211 | 6.1233 | 12.9662 | 77.124 | 9.64 | 44907.2148 | 75499.0547 |
| 4000 | 0.3232 | 49920.5820 | 60082.1211 | 6.1233 | 12.9824 | 77.027 | 9.628 | 44907.2148 | 75499.0547 |
| 4500 | 0.3636 | 49920.5820 | 60082.1211 | 6.1233 | 12.9965 | 76.944 | 9.618 | 44907.2148 | 75499.0547 |
| 5000 | 0.4040 | 49928.3438 | 60082.1211 | 6.1235 | 13.1112 | 76.271 | 9.534 | 44922.0352 | 75499.0547 |
| 5500 | 0.4444 | 49928.3438 | 60082.1211 | 6.1235 | 13.1865 | 75.835 | 9.479 | 44922.0352 | 75499.0547 |
| 6000 | 0.4848 | 49928.3438 | 60082.1211 | 6.1235 | 13.0376 | 76.701 | 9.588 | 44922.0352 | 75499.0547 |
| 6500 | 0.5253 | 49928.3438 | 60082.1211 | 6.1235 | 12.9934 | 76.962 | 9.62 | 44922.0352 | 75499.0547 |
| 7000 | 0.5657 | 49928.3438 | 60082.1211 | 6.1235 | 12.9741 | 77.077 | 9.635 | 44922.0352 | 75499.0547 |
| 7500 | 0.6061 | 49928.3438 | 60082.1211 | 6.1235 | 13.0011 | 76.917 | 9.615 | 44922.0352 | 75499.0547 |
| 8000 | 0.6465 | 49928.3438 | 60082.1211 | 6.1235 | 13.021 | 76.799 | 9.6 | 44922.0352 | 75499.0547 |
| 8500 | 0.6869 | 49928.3438 | 60082.1211 | 6.1235 | 13.023 | 76.787 | 9.598 | 44922.0352 | 75499.0547 |
| 9000 | 0.7273 | 49928.3438 | 60082.1211 | 6.1235 | 12.9717 | 77.091 | 9.636 | 44922.0352 | 75499.0547 |
| 9500 | 0.7677 | 49928.3438 | 60082.1211 | 6.1235 | 13.0526 | 76.613 | 9.577 | 44922.0352 | 75499.0547 |
| 10000 | 0.8081 | 49928.3438 | 60082.1211 | 6.1235 | 12.9964 | 76.944 | 9.618 | 44922.0352 | 75499.0547 |
| 10500 | 0.8485 | 49928.3438 | 60082.1211 | 6.1235 | 12.9662 | 77.123 | 9.64 | 44922.0352 | 75499.0547 |
| 11000 | 0.8889 | 49928.3438 | 60082.1211 | 6.1235 | 12.9957 | 76.949 | 9.619 | 44922.0352 | 75499.0547 |
| 11500 | 0.9293 | 49928.3438 | 60082.1211 | 6.1235 | 12.9518 | 77.209 | 9.651 | 44922.0352 | 75499.0547 |
| 12000 | 0.9697 | 49928.3438 | 60082.1211 | 6.1235 | 12.9583 | 77.171 | 9.646 | 44922.0352 | 75499.0547 |
| 12375 | 1.0 | 49928.3438 | 60082.1211 | 6.1235 | 13.0092 | 76.869 | 9.609 | 44922.0352 | 75499.0547 |

### Framework versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.21.0