distily_bench_obj_cross_v2.7

This student model is distilled from the teacher model roneneldan/TinyStories-33M using the dataset (unspecified).

The Distily library was used for this distillation.

It achieves the following results on the evaluation set:

• eval_enwikippl: 3887.9170
• eval_frwikippl: 50974.8398
• eval_zhwikippl: 83822.7812
• eval_tinystoriesppl: 1011.5359
• eval_loss: 4.8822
• eval_runtime: 6.5175
• eval_samples_per_second: 76.716
• eval_steps_per_second: 9.666

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
• train_embeddings: True
• learning_rate: 0.0004
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1.0

Resource Usage

Peak GPU Memory: 6.6058 GB

Eval-Phase Metrics

step	epoch	enwikippl	frwikippl	loss	runtime	samples_per_second	steps_per_second	tinystoriesppl	zhwikippl
teacher eval		169.9865	47377.9414					3.9789	4998.1294
0	0	23232.2363	111004.0469	6.4068	6.5373	76.484	9.637	9550.5166	102446.0156
500	0.0101	3972.8616	51053.8984	4.8833	6.5063	76.848	9.683	1038.9088	84361.2188
1000	0.0202	3908.7549	50974.8398	4.8822	6.5275	76.599	9.651	1016.2294	83867.4844
1500	0.0303	3887.6167	50974.8398	4.8822	6.5158	76.737	9.669	1010.1989	83867.4844
2000	0.0404	3873.1880	50989.1836	4.8822	6.5149	76.747	9.67	1005.8657	83822.7812
2500	0.0505	3897.8667	50974.8398	4.8822	6.5124	76.777	9.674	1014.4666	83822.7812
3000	0.0606	3906.9363	50974.8398	4.8822	6.5204	76.682	9.662	1016.0612	83822.7812
3500	0.0707	3893.6423	50974.8398	4.8822	6.5097	76.809	9.678	1012.0377	83822.7812
4000	0.0808	3894.2476	50974.8398	4.8822	6.511	76.793	9.676	1013.2932	83822.7812
4500	0.0909	3917.8462	51010.7305	4.8822	6.5167	76.726	9.668	1019.4268	83867.4844
5000	0.1010	3912.3875	51003.5820	4.8822	6.5075	76.834	9.681	1017.7424	83867.4844
5500	0.1111	3903.9119	50974.8398	4.8822	6.5226	76.656	9.659	1015.6412	83822.7812
6000	0.1212	3912.3875	51003.5820	4.8822	6.5212	76.673	9.661	1017.4061	83867.4844
6500	0.1313	3936.0994	51068.2656	4.8822	6.4984	76.942	9.695	1025.1731	83957.0312
7000	0.1414	3922.1003	51053.8984	4.8822	6.5149	76.747	9.67	1020.9445	83912.2812
7500	0.1515	3914.2085	51010.7305	4.8822	6.5141	76.757	9.671	1018.9213	83867.4844
8000	0.1616	3907.5435	50974.8398	4.8822	6.5148	76.748	9.67	1016.2294	83867.4844
8500	0.1717	3913.6001	51003.5820	4.8822	6.5271	76.603	9.652	1018.0792	83867.4844
9000	0.1818	3897.8667	50974.8398	4.8822	6.5537	76.293	9.613	1014.1313	83822.7812
9500	0.1919	3887.9170	50974.8398	4.8822	6.5175	76.716	9.666	1011.5359	83822.7812
10000	0.2020	3882.1963	50960.4531	4.8822	6.5134	76.765	9.672	1008.7803	83822.7812
10500	0.2121	3912.3875	51003.5820	4.8822	6.5102	76.802	9.677	1017.5745	83867.4844
11000	0.2222	3938.5400	51068.2656	4.8822	6.4952	76.98	9.699	1025.5123	83957.0312
11500	0.2323	3939.7610	51068.2656	4.8822	6.4945	76.988	9.7	1028.0588	84136.4141
12000	0.2424	3951.9873	51097.0547	4.8822	6.5097	76.809	9.678	1029.9302	84136.4141
12500	0.2525	3922.1003	51053.8984	4.8822	6.5036	76.881	9.687	1021.7891	83912.2812
13000	0.2626	3897.8667	50974.8398	4.8822	6.5248	76.631	9.655	1014.6344	83822.7812
13500	0.2727	3887.6167	50974.8398	4.8822	6.5036	76.881	9.687	1010.3655	83822.7812
14000	0.2828	3867.7913	50989.1836	4.8822	6.5118	76.783	9.675	1001.6337	83778.0312
14500	0.2929	3860.6052	50989.1836	4.8822	6.5064	76.847	9.683	999.4833	83778.0312
15000	0.3030	3873.1880	50960.4531	4.8822	6.5104	76.8	9.677	1005.8657	83822.7812
15500	0.3131	3884.0034	50960.4531	4.8822	6.5186	76.704	9.665	1009.1969	83822.7812
16000	0.3232	3876.7874	50989.1836	4.8822	6.5138	76.76	9.672	1006.6141	83822.7812
16500	0.3333	3902.7021	50974.8398	4.8822	6.5538	76.291	9.613	1015.1377	83822.7812
17000	0.3434	3894.2476	50974.8398	4.8822	6.5508	76.326	9.617	1013.2932	83822.7812
17500	0.3535	3895.4548	50974.8398	4.8822	6.5145	76.752	9.671	1013.4609	83822.7812
18000	0.3636	3892.4358	50974.8398	4.8822	6.5208	76.678	9.661	1011.7029	83822.7812
18500	0.3737	3895.4548	50974.8398	4.8822	6.506	76.852	9.683	1013.4609	83822.7812
19000	0.3838	3922.1003	51053.8984	4.8822	6.4973	76.955	9.696	1021.7891	83912.2812
19500	0.3939	3931.2227	51068.2656	4.8822	6.4889	77.055	9.709	1023.6490	83957.0312
20000	0.4040	3926.9622	51068.2656	4.8822	6.4907	77.034	9.706	1023.3103	83912.2812
20500	0.4141	3922.1003	51053.8984	4.8822	6.4861	77.088	9.713	1021.7891	83912.2812
21000	0.4242	3922.1003	51053.8984	4.8822	6.4983	76.943	9.695	1021.7891	83912.2812
21500	0.4343	3931.2227	51068.2656	4.8822	6.509	76.817	9.679	1024.8344	83957.0312
22000	0.4444	3938.5400	51068.2656	4.8822	6.5034	76.883	9.687	1026.6998	84091.5703
22500	0.4545	3931.2227	51068.2656	4.8822	6.5235	76.646	9.657	1024.8344	83957.0312
23000	0.4646	3922.1003	51053.8984	4.8822	6.5476	76.363	9.622	1021.4509	83867.4844
23500	0.4747	3920.8850	51053.8984	4.8822	6.4853	77.098	9.714	1020.6072	83867.4844
24000	0.4848	3919.6699	51053.8984	4.8822	6.4956	76.975	9.699	1020.4384	83867.4844
24500	0.4949	3921.4907	51053.8984	4.8822	6.5036	76.88	9.687	1020.7761	83867.4844
25000	0.5051	3919.6699	51053.8984	4.8822	6.4891	77.052	9.709	1020.6072	83867.4844
25500	0.5152	3922.1003	51053.8984	4.8822	6.4927	77.01	9.703	1020.9445	83867.4844
26000	0.5253	3926.9622	51068.2656	4.8822	6.4956	76.975	9.699	1022.4650	83912.2812
26500	0.5354	3931.2227	51068.2656	4.8822	6.5008	76.914	9.691	1024.1567	83957.0312
27000	0.5455	3926.9622	51068.2656	4.8822	6.5048	76.867	9.685	1022.2959	83912.2812
27500	0.5556	3922.1003	51053.8984	4.8822	6.526	76.617	9.654	1021.7891	83912.2812
28000	0.5657	3921.4907	51053.8984	4.8822	6.5337	76.527	9.642	1020.7761	83867.4844
28500	0.5758	3917.8462	51053.8984	4.8822	6.4962	76.969	9.698	1019.5950	83867.4844
29000	0.5859	3917.8462	51039.4883	4.8822	6.4943	76.99	9.701	1019.4268	83867.4844
29500	0.5960	3919.6699	51053.8984	4.8822	6.5069	76.841	9.682	1020.4384	83867.4844
30000	0.6061	3919.6699	51053.8984	4.8822	6.5033	76.884	9.687	1020.4384	83867.4844
30500	0.6162	3917.8462	51039.4883	4.8822	6.4958	76.972	9.699	1019.4268	83867.4844
31000	0.6263	3917.8462	51039.4883	4.8822	6.4827	77.129	9.718	1019.4268	83867.4844
31500	0.6364	3917.8462	51039.4883	4.8822	6.495	76.982	9.7	1019.4268	83867.4844
32000	0.6465	3918.4551	51053.8984	4.8822	6.5193	76.696	9.664	1019.5950	83867.4844
32500	0.6566	3921.4907	51053.8984	4.8822	6.5213	76.671	9.661	1020.7761	83867.4844
33000	0.6667	3922.1003	51053.8984	4.8822	6.5	76.923	9.692	1021.4509	83867.4844
33500	0.6768	3922.1003	51053.8984	4.8822	6.5038	76.878	9.687	1021.7891	83912.2812
34000	0.6869	3922.1003	51053.8984	4.8822	6.5965	75.798	9.551	1021.6200	83867.4844
34500	0.6970	3922.1003	51053.8984	4.8822	6.4926	77.01	9.703	1020.7761	83867.4844
35000	0.7071	3921.4907	51053.8984	4.8822	6.5061	76.851	9.683	1020.7761	83867.4844
35500	0.7172	3921.4907	51053.8984	4.8822	6.5289	76.582	9.649	1020.7761	83867.4844
36000	0.7273	3922.1003	51053.8984	4.8822	6.5582	76.24	9.606	1021.4509	83867.4844
36500	0.7374	3922.1003	51053.8984	4.8822	6.5354	76.506	9.64	1021.7891	83912.2812
37000	0.7475	3924.5286	51053.8984	4.8822	6.5215	76.669	9.66	1021.7891	83912.2812
37500	0.7576	3926.9622	51053.8984	4.8822	6.5007	76.915	9.691	1021.7891	83912.2812
38000	0.7677	3924.5286	51053.8984	4.8822	6.5068	76.842	9.682	1021.7891	83912.2812
38500	0.7778	3922.1003	51053.8984	4.8822	6.5229	76.653	9.658	1021.6200	83867.4844
39000	0.7879	3922.1003	51053.8984	4.8822	6.5165	76.728	9.668	1020.9445	83867.4844
39500	0.7980	3920.8850	51053.8984	4.8822	6.5119	76.782	9.675	1020.6072	83867.4844
40000	0.8081	3920.8850	51053.8984	4.8822	6.5092	76.814	9.679	1020.6072	83867.4844
40500	0.8182	3920.8850	51053.8984	4.8822	6.5191	76.697	9.664	1020.7761	83867.4844
41000	0.8283	3920.8850	51053.8984	4.8822	6.5316	76.551	9.645	1020.7761	83867.4844
41500	0.8384	3920.8850	51053.8984	4.8822	6.5046	76.869	9.685	1020.7761	83867.4844
42000	0.8485	3920.8850	51053.8984	4.8822	6.5038	76.878	9.687	1020.6072	83867.4844
42500	0.8586	3920.8850	51053.8984	4.8822	6.5215	76.669	9.66	1020.6072	83867.4844
43000	0.8687	3920.8850	51053.8984	4.8822	6.5049	76.865	9.685	1020.6072	83867.4844
43500	0.8788	3920.8850	51053.8984	4.8822	6.5074	76.836	9.681	1020.6072	83867.4844
44000	0.8889	3920.8850	51053.8984	4.8822	6.4973	76.956	9.696	1020.6072	83867.4844
44500	0.8990	3920.8850	51053.8984	4.8822	6.529	76.581	9.649	1020.6072	83867.4844
45000	0.9091	3920.8850	51053.8984	4.8822	6.5231	76.651	9.658	1020.6072	83867.4844
45500	0.9192	3920.8850	51053.8984	4.8822	6.5386	76.469	9.635	1020.6072	83867.4844
46000	0.9293	3920.8850	51053.8984	4.8822	6.5266	76.61	9.653	1020.6072	83867.4844
46500	0.9394	3920.8850	51053.8984	4.8822	6.4999	76.924	9.692	1020.6072	83867.4844
47000	0.9495	3920.8850	51053.8984	4.8822	6.5248	76.63	9.655	1020.6072	83867.4844
47500	0.9596	3920.8850	51053.8984	4.8822	6.5217	76.668	9.66	1020.6072	83867.4844
48000	0.9697	3920.8850	51053.8984	4.8822	6.5043	76.872	9.686	1020.6072	83867.4844
48500	0.9798	3920.8850	51053.8984	4.8822	6.4865	77.084	9.713	1020.6072	83867.4844
49000	0.9899	3920.8850	51053.8984	4.8822	6.5203	76.684	9.662	1020.6072	83867.4844
49500	1.0	3920.8850	51053.8984	4.8822	6.6387	75.315	9.49	1020.6072	83867.4844

Framework versions

• Distily 0.2.0
• Transformers 4.44.0
• Pytorch 2.3.0
• Datasets 2.21.0