| INFO: 2024-10-28 13:15:15,094: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] | |
| INFO: 2024-10-28 13:15:15,094: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:15:15,094: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:15:16,695: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] | |
| INFO: 2024-10-28 13:15:16,695: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:15:16,695: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:15:18,943: llmtf.base.darumeru/PARus: Loading Dataset: 2.25s | |
| INFO: 2024-10-28 13:15:19,297: llmtf.base.darumeru/MultiQ: Loading Dataset: 4.20s | |
| INFO: 2024-10-28 13:15:22,318: llmtf.base.darumeru/PARus: Processing Dataset: 3.37s | |
| INFO: 2024-10-28 13:15:22,318: llmtf.base.darumeru/PARus: Results for darumeru/PARus: | |
| INFO: 2024-10-28 13:15:22,329: llmtf.base.darumeru/PARus: {'acc': 0.78} | |
| INFO: 2024-10-28 13:15:22,330: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:15:22,330: llmtf.base.evaluator: | |
| mean darumeru/PARus | |
| 0.780 0.780 | |
| INFO: 2024-10-28 13:15:30,304: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] | |
| INFO: 2024-10-28 13:15:30,304: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:15:30,304: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:15:33,637: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 3.33s | |
| INFO: 2024-10-28 13:16:05,173: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 31.54s | |
| INFO: 2024-10-28 13:16:05,173: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: | |
| INFO: 2024-10-28 13:16:05,184: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.8256013745704467, 'f1_macro': 0.8262484506706507} | |
| INFO: 2024-10-28 13:16:05,191: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:16:05,192: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/ruOpenBookQA | |
| 0.803 0.780 0.826 | |
| INFO: 2024-10-28 13:16:13,923: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] | |
| INFO: 2024-10-28 13:16:13,923: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:16:13,923: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:16:16,429: llmtf.base.darumeru/RWSD: Loading Dataset: 2.51s | |
| INFO: 2024-10-28 13:16:22,246: llmtf.base.darumeru/RWSD: Processing Dataset: 5.82s | |
| INFO: 2024-10-28 13:16:22,246: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: | |
| INFO: 2024-10-28 13:16:22,247: llmtf.base.darumeru/RWSD: {'acc': 0.5441176470588235} | |
| INFO: 2024-10-28 13:16:22,248: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:16:22,249: llmtf.base.evaluator: | |
| mean darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA | |
| 0.717 0.780 0.544 0.826 | |
| INFO: 2024-10-28 13:16:31,348: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] | |
| INFO: 2024-10-28 13:16:31,348: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:16:31,348: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:18:38,554: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 127.21s | |
| INFO: 2024-10-28 13:20:06,478: llmtf.base.darumeru/MultiQ: Processing Dataset: 287.18s | |
| INFO: 2024-10-28 13:20:06,479: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: | |
| INFO: 2024-10-28 13:20:06,480: llmtf.base.darumeru/MultiQ: {'f1': 0.2503859074384594, 'em': 0.14531548757170173} | |
| INFO: 2024-10-28 13:20:06,488: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:20:06,489: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA | |
| 0.587 0.198 0.780 0.544 0.826 | |
| INFO: 2024-10-28 13:20:15,334: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] | |
| INFO: 2024-10-28 13:20:15,335: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:20:15,335: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:20:18,179: llmtf.base.darumeru/RCB: Loading Dataset: 2.84s | |
| INFO: 2024-10-28 13:20:23,505: llmtf.base.darumeru/RCB: Processing Dataset: 5.33s | |
| INFO: 2024-10-28 13:20:23,506: llmtf.base.darumeru/RCB: Results for darumeru/RCB: | |
| INFO: 2024-10-28 13:20:23,510: llmtf.base.darumeru/RCB: {'acc': 0.5863636363636363, 'f1_macro': 0.5032640286161413} | |
| INFO: 2024-10-28 13:20:23,511: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:20:23,512: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA | |
| 0.579 0.198 0.780 0.545 0.544 0.826 | |
| INFO: 2024-10-28 13:20:32,046: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] | |
| INFO: 2024-10-28 13:20:32,046: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:20:32,046: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:20:34,403: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.36s | |
| INFO: 2024-10-28 13:20:36,969: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.57s | |
| INFO: 2024-10-28 13:20:36,969: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: | |
| INFO: 2024-10-28 13:20:36,972: llmtf.base.darumeru/ruWorldTree: {'acc': 0.9047619047619048, 'f1_macro': 0.9038817229146561} | |
| INFO: 2024-10-28 13:20:36,972: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:20:36,972: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree | |
| 0.633 0.198 0.780 0.545 0.544 0.826 0.904 | |
| INFO: 2024-10-28 13:20:45,488: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] | |
| INFO: 2024-10-28 13:20:45,488: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:20:45,488: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:20:59,443: llmtf.base.daru/treewayextractive: Loading Dataset: 13.95s | |
| INFO: 2024-10-28 13:23:49,533: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 310.98s | |
| INFO: 2024-10-28 13:23:49,533: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: | |
| INFO: 2024-10-28 13:23:49,597: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| abstract_algebra 0.480000 | |
| anatomy 0.562963 | |
| astronomy 0.769737 | |
| business_ethics 0.640000 | |
| clinical_knowledge 0.671698 | |
| college_biology 0.673611 | |
| college_chemistry 0.470000 | |
| college_computer_science 0.670000 | |
| college_mathematics 0.440000 | |
| college_medicine 0.589595 | |
| college_physics 0.480392 | |
| computer_security 0.720000 | |
| conceptual_physics 0.655319 | |
| econometrics 0.482456 | |
| electrical_engineering 0.606897 | |
| elementary_mathematics 0.616402 | |
| formal_logic 0.428571 | |
| global_facts 0.370000 | |
| high_school_biology 0.809677 | |
| high_school_chemistry 0.571429 | |
| high_school_computer_science 0.770000 | |
| high_school_european_history 0.751515 | |
| high_school_geography 0.782828 | |
| high_school_government_and_politics 0.725389 | |
| high_school_macroeconomics 0.658974 | |
| high_school_mathematics 0.525926 | |
| high_school_microeconomics 0.705882 | |
| high_school_physics 0.463576 | |
| high_school_psychology 0.796330 | |
| high_school_statistics 0.606481 | |
| high_school_us_history 0.779412 | |
| high_school_world_history 0.801688 | |
| human_aging 0.632287 | |
| human_sexuality 0.717557 | |
| international_law 0.743802 | |
| jurisprudence 0.675926 | |
| logical_fallacies 0.662577 | |
| machine_learning 0.482143 | |
| management 0.747573 | |
| marketing 0.816239 | |
| medical_genetics 0.650000 | |
| miscellaneous 0.711367 | |
| moral_disputes 0.627168 | |
| moral_scenarios 0.244693 | |
| nutrition 0.689542 | |
| philosophy 0.646302 | |
| prehistory 0.660494 | |
| professional_accounting 0.439716 | |
| professional_law 0.411343 | |
| professional_medicine 0.613971 | |
| professional_psychology 0.591503 | |
| public_relations 0.545455 | |
| security_studies 0.665306 | |
| sociology 0.736318 | |
| us_foreign_policy 0.800000 | |
| virology 0.500000 | |
| world_religions 0.760234 | |
| INFO: 2024-10-28 13:23:49,606: llmtf.base.nlpcoreteam/ruMMLU: metric | |
| subject | |
| STEM 0.600644 | |
| humanities 0.630286 | |
| other (business, health, misc.) 0.616782 | |
| social sciences 0.684000 | |
| INFO: 2024-10-28 13:23:49,611: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.6329281396317665} | |
| INFO: 2024-10-28 13:23:49,646: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:23:49,648: llmtf.base.evaluator: | |
| mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU | |
| 0.633 0.198 0.780 0.545 0.544 0.826 0.904 0.633 | |
| INFO: 2024-10-28 13:23:57,887: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] | |
| INFO: 2024-10-28 13:23:57,887: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:23:57,887: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:24:02,221: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.33s | |
| INFO: 2024-10-28 13:26:15,188: llmtf.base.daru/treewayextractive: Processing Dataset: 315.74s | |
| INFO: 2024-10-28 13:26:15,188: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: | |
| INFO: 2024-10-28 13:26:15,447: llmtf.base.daru/treewayextractive: {'r-prec': 0.40380281385281386} | |
| INFO: 2024-10-28 13:26:15,501: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:26:15,503: llmtf.base.evaluator: | |
| mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU | |
| 0.604 0.404 0.198 0.780 0.545 0.544 0.826 0.904 0.633 | |
| INFO: 2024-10-28 13:26:24,206: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] | |
| INFO: 2024-10-28 13:26:24,207: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:26:24,207: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:26:48,154: llmtf.base.daru/treewayabstractive: Processing Dataset: 165.93s | |
| INFO: 2024-10-28 13:26:48,154: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: | |
| INFO: 2024-10-28 13:26:48,155: llmtf.base.daru/treewayabstractive: {'rouge1': 0.3489002151166006, 'rouge2': 0.12404569962254197} | |
| INFO: 2024-10-28 13:26:48,156: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:26:48,157: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU | |
| 0.563 0.236 0.404 0.198 0.780 0.545 0.544 0.826 0.904 0.633 | |
| INFO: 2024-10-28 13:28:23,832: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 119.62s | |
| INFO: 2024-10-28 13:33:05,781: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 281.95s | |
| INFO: 2024-10-28 13:33:05,781: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: | |
| INFO: 2024-10-28 13:33:05,844: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| abstract_algebra 0.450000 | |
| anatomy 0.725926 | |
| astronomy 0.861842 | |
| business_ethics 0.750000 | |
| clinical_knowledge 0.762264 | |
| college_biology 0.854167 | |
| college_chemistry 0.510000 | |
| college_computer_science 0.720000 | |
| college_mathematics 0.470000 | |
| college_medicine 0.699422 | |
| college_physics 0.509804 | |
| computer_security 0.770000 | |
| conceptual_physics 0.706383 | |
| econometrics 0.605263 | |
| electrical_engineering 0.696552 | |
| elementary_mathematics 0.666667 | |
| formal_logic 0.492063 | |
| global_facts 0.420000 | |
| high_school_biology 0.861290 | |
| high_school_chemistry 0.620690 | |
| high_school_computer_science 0.840000 | |
| high_school_european_history 0.824242 | |
| high_school_geography 0.873737 | |
| high_school_government_and_politics 0.927461 | |
| high_school_macroeconomics 0.761538 | |
| high_school_mathematics 0.566667 | |
| high_school_microeconomics 0.873950 | |
| high_school_physics 0.582781 | |
| high_school_psychology 0.888073 | |
| high_school_statistics 0.708333 | |
| high_school_us_history 0.838235 | |
| high_school_world_history 0.860759 | |
| human_aging 0.762332 | |
| human_sexuality 0.786260 | |
| international_law 0.809917 | |
| jurisprudence 0.796296 | |
| logical_fallacies 0.828221 | |
| machine_learning 0.526786 | |
| management 0.854369 | |
| marketing 0.914530 | |
| medical_genetics 0.810000 | |
| miscellaneous 0.848020 | |
| moral_disputes 0.736994 | |
| moral_scenarios 0.459218 | |
| nutrition 0.797386 | |
| philosophy 0.723473 | |
| prehistory 0.805556 | |
| professional_accounting 0.556738 | |
| professional_law 0.507823 | |
| professional_medicine 0.742647 | |
| professional_psychology 0.750000 | |
| public_relations 0.636364 | |
| security_studies 0.759184 | |
| sociology 0.845771 | |
| us_foreign_policy 0.850000 | |
| virology 0.506024 | |
| world_religions 0.853801 | |
| INFO: 2024-10-28 13:33:05,852: llmtf.base.nlpcoreteam/enMMLU: metric | |
| subject | |
| STEM 0.662331 | |
| humanities 0.733585 | |
| other (business, health, misc.) 0.724976 | |
| social sciences 0.796467 | |
| INFO: 2024-10-28 13:33:05,857: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.7293395108036221} | |
| INFO: 2024-10-28 13:33:05,908: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:33:05,910: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU | |
| 0.580 0.236 0.404 0.198 0.780 0.545 0.544 0.826 0.904 0.729 0.633 | |
| INFO: 2024-10-28 13:33:14,562: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] | |
| INFO: 2024-10-28 13:33:14,562: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] | |
| INFO: 2024-10-28 13:33:14,562: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] | |
| INFO: 2024-10-28 13:33:17,057: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.49s | |
| INFO: 2024-10-28 13:35:21,669: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 124.61s | |
| INFO: 2024-10-28 13:35:21,670: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: | |
| INFO: 2024-10-28 13:35:21,670: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.9953318595732386, 'len': 0.9990656928305265, 'lcs': 1.0} | |
| INFO: 2024-10-28 13:35:21,671: llmtf.base.evaluator: Ended eval | |
| INFO: 2024-10-28 13:35:21,672: llmtf.base.evaluator: | |
| mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU | |
| 0.618 0.236 0.404 0.198 0.780 0.545 0.544 1.000 0.826 0.904 0.729 0.633 | |