sembr2023-bert-small

This model is a fine-tuned version of prajjwal1/bert-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2324
• Precision: 0.7915
• Recall: 0.8418
• F1: 0.8159
• Iou: 0.6890
• Accuracy: 0.9651
• Balanced Accuracy: 0.9097
• Overall Accuracy: 0.9481

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 64
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• training_steps: 1000

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Iou	Accuracy	Balanced Accuracy	Overall Accuracy
0.4134	0.06	10	0.4107	0	0.0	0.0	0.0	0.9080	0.5	0.9080
0.371	0.12	20	0.3698	0	0.0	0.0	0.0	0.9080	0.5	0.9080
0.2913	0.18	30	0.2672	0.8443	0.4167	0.5580	0.3870	0.9393	0.7045	0.9283
0.2315	0.24	40	0.2184	0.8043	0.6761	0.7346	0.5806	0.9551	0.8297	0.9364
0.1693	0.3	50	0.2021	0.8064	0.7375	0.7704	0.6265	0.9596	0.8598	0.9396
0.1812	0.36	60	0.1869	0.8727	0.6847	0.7674	0.6225	0.9618	0.8373	0.9437
0.1745	0.42	70	0.1855	0.8021	0.7744	0.7880	0.6502	0.9617	0.8775	0.9421
0.1577	0.48	80	0.1817	0.8207	0.7641	0.7914	0.6548	0.9630	0.8736	0.9431
0.1458	0.55	90	0.1763	0.8183	0.7869	0.8023	0.6698	0.9643	0.8846	0.9449
0.1343	0.61	100	0.1772	0.8721	0.7372	0.7990	0.6652	0.9659	0.8631	0.9477
0.1442	0.67	110	0.1647	0.8388	0.7795	0.8081	0.6779	0.9659	0.8822	0.9483
0.1104	0.73	120	0.1678	0.8488	0.7679	0.8063	0.6755	0.9661	0.8770	0.9479
0.1089	0.79	130	0.1745	0.7882	0.8262	0.8068	0.6761	0.9636	0.9019	0.9434
0.1437	0.85	140	0.1768	0.7970	0.8206	0.8086	0.6787	0.9643	0.8997	0.9440
0.1104	0.91	150	0.1710	0.7961	0.8275	0.8115	0.6828	0.9646	0.9030	0.9446
0.0941	0.97	160	0.1647	0.8007	0.8167	0.8086	0.6787	0.9644	0.8980	0.9456
0.1146	1.03	170	0.1744	0.8026	0.8250	0.8136	0.6858	0.9652	0.9022	0.9456
0.0982	1.09	180	0.1636	0.8175	0.8191	0.8183	0.6925	0.9666	0.9003	0.9468
0.0875	1.15	190	0.1653	0.8305	0.8064	0.8183	0.6924	0.9671	0.8948	0.9476
0.0962	1.21	200	0.1610	0.8340	0.8076	0.8206	0.6958	0.9675	0.8957	0.9490
0.084	1.27	210	0.1671	0.8232	0.8177	0.8204	0.6955	0.9671	0.9000	0.9476
0.07	1.33	220	0.1665	0.7909	0.8545	0.8215	0.6971	0.9658	0.9158	0.9454
0.0785	1.39	230	0.1612	0.8411	0.8004	0.8202	0.6953	0.9677	0.8925	0.9496
0.0712	1.45	240	0.1638	0.8251	0.8161	0.8205	0.6957	0.9672	0.8993	0.9491
0.0683	1.52	250	0.1823	0.8097	0.8262	0.8179	0.6919	0.9662	0.9033	0.9463
0.0694	1.58	260	0.1717	0.8028	0.8408	0.8214	0.6969	0.9664	0.9099	0.9474
0.0809	1.64	270	0.1681	0.8304	0.8102	0.8202	0.6952	0.9673	0.8967	0.9491
0.0586	1.7	280	0.1811	0.8096	0.8391	0.8241	0.7008	0.9671	0.9096	0.9478
0.069	1.76	290	0.1855	0.8088	0.8284	0.8185	0.6928	0.9662	0.9043	0.9478
0.0739	1.82	300	0.1876	0.8148	0.8209	0.8178	0.6918	0.9664	0.9010	0.9476
0.0691	1.88	310	0.1741	0.8173	0.8206	0.8190	0.6934	0.9666	0.9010	0.9485
0.0728	1.94	320	0.1765	0.7941	0.8346	0.8139	0.6861	0.9649	0.9064	0.9469
0.0585	2.0	330	0.1800	0.8118	0.8166	0.8142	0.6866	0.9657	0.8987	0.9483
0.0602	2.06	340	0.1930	0.7969	0.8366	0.8162	0.6895	0.9654	0.9075	0.9461
0.0557	2.12	350	0.1832	0.7915	0.8401	0.8151	0.6879	0.9649	0.9089	0.9472
0.0491	2.18	360	0.1914	0.8131	0.8136	0.8134	0.6854	0.9657	0.8973	0.9489
0.0413	2.24	370	0.2116	0.7989	0.8288	0.8136	0.6857	0.9651	0.9038	0.9463
0.051	2.3	380	0.2073	0.7864	0.8454	0.8148	0.6875	0.9647	0.9111	0.9460
0.0529	2.36	390	0.1923	0.8103	0.8278	0.8190	0.6934	0.9663	0.9041	0.9496
0.0469	2.42	400	0.1808	0.8131	0.8217	0.8173	0.6911	0.9662	0.9013	0.9497
0.0579	2.48	410	0.2053	0.7795	0.8493	0.8129	0.6848	0.9640	0.9125	0.9464
0.0494	2.55	420	0.1953	0.7872	0.8457	0.8154	0.6883	0.9648	0.9113	0.9471
0.0468	2.61	430	0.1972	0.8064	0.8182	0.8123	0.6839	0.9652	0.8992	0.9488
0.0545	2.67	440	0.2116	0.7774	0.8455	0.8100	0.6807	0.9635	0.9105	0.9458
0.0544	2.73	450	0.1954	0.7868	0.8455	0.8151	0.6879	0.9647	0.9111	0.9472
0.044	2.79	460	0.2046	0.8149	0.8203	0.8175	0.6914	0.9663	0.9007	0.9491
0.0468	2.85	470	0.2036	0.8031	0.8321	0.8174	0.6911	0.9658	0.9057	0.9483
0.0457	2.91	480	0.1998	0.7923	0.8377	0.8144	0.6869	0.9649	0.9077	0.9479
0.0435	2.97	490	0.2077	0.7864	0.8432	0.8138	0.6860	0.9645	0.9100	0.9475
0.0489	3.03	500	0.2067	0.7933	0.8339	0.8131	0.6850	0.9647	0.9059	0.9478
0.0472	3.09	510	0.2204	0.7883	0.8464	0.8163	0.6896	0.9650	0.9117	0.9475
0.0469	3.15	520	0.2209	0.7821	0.8470	0.8132	0.6853	0.9642	0.9115	0.9467
0.0384	3.21	530	0.2147	0.7923	0.8367	0.8139	0.6862	0.9648	0.9072	0.9479
0.0322	3.27	540	0.2215	0.7842	0.8489	0.8153	0.6881	0.9646	0.9126	0.9475
0.0429	3.33	550	0.2184	0.7743	0.8504	0.8106	0.6815	0.9634	0.9127	0.9463
0.0348	3.39	560	0.2293	0.7642	0.8594	0.8090	0.6792	0.9627	0.9163	0.9451
0.0365	3.45	570	0.2221	0.7922	0.8411	0.8159	0.6891	0.9651	0.9094	0.9477
0.0374	3.52	580	0.2175	0.7917	0.8382	0.8143	0.6868	0.9648	0.9079	0.9479
0.0413	3.58	590	0.2111	0.8122	0.8243	0.8182	0.6924	0.9663	0.9025	0.9499
0.0362	3.64	600	0.2183	0.7883	0.8404	0.8135	0.6856	0.9646	0.9088	0.9479
0.0352	3.7	610	0.2124	0.8005	0.8340	0.8169	0.6905	0.9656	0.9065	0.9487
0.0301	3.76	620	0.2145	0.7993	0.8369	0.8177	0.6916	0.9657	0.9078	0.9488
0.0399	3.82	630	0.2188	0.8018	0.8318	0.8166	0.6900	0.9656	0.9055	0.9485
0.0366	3.88	640	0.2211	0.7969	0.8346	0.8153	0.6882	0.9652	0.9066	0.9478
0.0289	3.94	650	0.2201	0.7850	0.8468	0.8147	0.6874	0.9646	0.9116	0.9475
0.0367	4.0	660	0.2280	0.7859	0.8437	0.8138	0.6860	0.9645	0.9102	0.9475
0.0362	4.06	670	0.2226	0.7785	0.8502	0.8128	0.6846	0.9640	0.9128	0.9469
0.0376	4.12	680	0.2213	0.8006	0.8317	0.8159	0.6890	0.9655	0.9054	0.9490
0.0294	4.18	690	0.2212	0.8052	0.8271	0.8160	0.6892	0.9657	0.9034	0.9492
0.0318	4.24	700	0.2254	0.7874	0.8420	0.8138	0.6860	0.9646	0.9095	0.9477
0.0359	4.3	710	0.2250	0.7899	0.8432	0.8157	0.6887	0.9649	0.9102	0.9479
0.034	4.36	720	0.2264	0.7985	0.8380	0.8178	0.6917	0.9656	0.9083	0.9489
0.0334	4.42	730	0.2308	0.7871	0.8436	0.8144	0.6869	0.9646	0.9102	0.9475
0.036	4.48	740	0.2250	0.7936	0.8404	0.8163	0.6896	0.9652	0.9091	0.9485
0.0257	4.55	750	0.2267	0.7861	0.8456	0.8148	0.6874	0.9646	0.9112	0.9479
0.0354	4.61	760	0.2288	0.7943	0.8401	0.8166	0.6900	0.9653	0.9090	0.9484
0.0373	4.67	770	0.2320	0.7828	0.8471	0.8137	0.6859	0.9643	0.9117	0.9470
0.0272	4.73	780	0.2250	0.7994	0.8354	0.8170	0.6906	0.9656	0.9071	0.9487
0.034	4.79	790	0.2339	0.7861	0.8450	0.8145	0.6870	0.9646	0.9108	0.9473
0.0294	4.85	800	0.2262	0.7972	0.8381	0.8171	0.6908	0.9655	0.9082	0.9486
0.0353	4.91	810	0.2337	0.7833	0.8473	0.8140	0.6864	0.9644	0.9118	0.9472
0.0337	4.97	820	0.2273	0.7973	0.8372	0.8168	0.6903	0.9655	0.9078	0.9485
0.0309	5.03	830	0.2318	0.7917	0.8413	0.8157	0.6888	0.9650	0.9094	0.9481
0.026	5.09	840	0.2327	0.7932	0.8418	0.8168	0.6903	0.9653	0.9098	0.9483
0.0271	5.15	850	0.2317	0.7887	0.8459	0.8163	0.6896	0.9650	0.9115	0.9479
0.0352	5.21	860	0.2344	0.7914	0.8427	0.8162	0.6895	0.9651	0.9101	0.9481
0.0268	5.27	870	0.2306	0.7931	0.8417	0.8166	0.6901	0.9652	0.9097	0.9484
0.0248	5.33	880	0.2309	0.7889	0.8438	0.8155	0.6884	0.9649	0.9105	0.9480
0.0331	5.39	890	0.2306	0.7895	0.8432	0.8154	0.6884	0.9649	0.9102	0.9480
0.0265	5.45	900	0.2322	0.7944	0.8401	0.8166	0.6901	0.9653	0.9091	0.9484
0.0352	5.52	910	0.2326	0.7922	0.8419	0.8163	0.6896	0.9651	0.9098	0.9482
0.0368	5.58	920	0.2313	0.7911	0.8424	0.8160	0.6891	0.9651	0.9099	0.9481
0.0315	5.64	930	0.2313	0.7917	0.8420	0.8161	0.6893	0.9651	0.9098	0.9482
0.0251	5.7	940	0.2324	0.7919	0.8409	0.8156	0.6887	0.9650	0.9093	0.9481
0.0331	5.76	950	0.2327	0.7913	0.8414	0.8156	0.6886	0.9650	0.9095	0.9481
0.0361	5.82	960	0.2327	0.7904	0.8423	0.8155	0.6885	0.9650	0.9098	0.9480
0.0362	5.88	970	0.2325	0.7909	0.8419	0.8156	0.6887	0.9650	0.9097	0.9481
0.031	5.94	980	0.2324	0.7914	0.8418	0.8158	0.6889	0.9650	0.9097	0.9481
0.0316	6.0	990	0.2324	0.7915	0.8418	0.8159	0.6890	0.9651	0.9097	0.9481
0.0232	6.06	1000	0.2324	0.7915	0.8418	0.8159	0.6890	0.9651	0.9097	0.9481

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.1
• Datasets 2.14.6
• Tokenizers 0.14.1