bart-pubhealth-expanded-hi-grad

This model is a fine-tuned version of facebook/bart-large on the clupubhealth dataset. It achieves the following results on the evaluation set:

• Loss: 2.0581
• Rouge1: 30.2592
• Rouge2: 11.7027
• Rougel: 24.1706
• Rougelsum: 24.3596
• Gen Len: 19.95

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 950
• total_train_batch_size: 15200
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
3.7893	0.49	2	2.3943	20.5187	5.4764	15.9378	16.2797	20.0
3.4045	0.98	4	2.1599	24.0858	7.8207	19.0412	19.1609	19.88
3.2488	1.47	6	2.1026	27.3466	9.369	21.1419	21.3136	19.865
3.1823	1.96	8	2.1324	28.825	9.6007	22.0963	22.3776	19.82
3.1263	2.44	10	2.1105	29.2694	10.5001	23.2842	23.5473	19.85
3.0834	2.93	12	2.0837	28.5975	10.2016	22.048	22.1341	19.915
3.0283	3.42	14	2.0773	28.5813	10.447	22.7456	22.8496	19.91
3.0301	3.91	16	2.0730	30.1049	11.4375	24.083	24.3045	19.945
2.9851	4.4	18	2.0775	29.2224	10.2722	22.7019	23.0038	19.95
2.9769	4.89	20	2.0777	29.6981	10.7044	23.2487	23.5232	19.96
2.9623	5.38	22	2.0711	29.0438	10.5105	23.1751	23.415	19.92
2.9421	5.87	24	2.0676	29.096	10.6599	23.1381	23.3765	19.985
2.9234	6.36	26	2.0646	29.6561	10.9096	23.2384	23.4265	19.985
2.9107	6.85	28	2.0616	29.7134	11.1686	23.272	23.4475	19.985
2.9077	7.33	30	2.0593	29.5055	11.0256	23.4406	23.6653	19.955
2.9072	7.82	32	2.0585	30.0504	11.433	23.9176	24.1728	19.95
2.8951	8.31	34	2.0583	29.9401	11.602	23.948	24.1323	19.95
2.8955	8.8	36	2.0584	30.1158	11.4745	24.0509	24.2465	19.94
2.8774	9.29	38	2.0582	30.0476	11.4465	23.8956	24.0527	19.945
2.8851	9.78	40	2.0581	30.2592	11.7027	24.1706	24.3596	19.95

Framework versions

• Transformers 4.31.0
• Pytorch 2.0.1+cu117
• Datasets 2.7.1
• Tokenizers 0.13.2