INFO: 2024-10-28 13:15:15,094: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] INFO: 2024-10-28 13:15:15,094: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:15:15,094: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:15:16,695: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] INFO: 2024-10-28 13:15:16,695: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:15:16,695: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:15:18,943: llmtf.base.darumeru/PARus: Loading Dataset: 2.25s INFO: 2024-10-28 13:15:19,297: llmtf.base.darumeru/MultiQ: Loading Dataset: 4.20s INFO: 2024-10-28 13:15:22,318: llmtf.base.darumeru/PARus: Processing Dataset: 3.37s INFO: 2024-10-28 13:15:22,318: llmtf.base.darumeru/PARus: Results for darumeru/PARus: INFO: 2024-10-28 13:15:22,329: llmtf.base.darumeru/PARus: {'acc': 0.78} INFO: 2024-10-28 13:15:22,330: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:15:22,330: llmtf.base.evaluator: mean darumeru/PARus 0.780 0.780 INFO: 2024-10-28 13:15:30,304: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] INFO: 2024-10-28 13:15:30,304: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:15:30,304: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:15:33,637: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 3.33s INFO: 2024-10-28 13:16:05,173: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 31.54s INFO: 2024-10-28 13:16:05,173: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: INFO: 2024-10-28 13:16:05,184: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.8256013745704467, 'f1_macro': 0.8262484506706507} INFO: 2024-10-28 13:16:05,191: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:16:05,192: llmtf.base.evaluator: mean darumeru/PARus darumeru/ruOpenBookQA 0.803 0.780 0.826 INFO: 2024-10-28 13:16:13,923: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] INFO: 2024-10-28 13:16:13,923: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:16:13,923: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:16:16,429: llmtf.base.darumeru/RWSD: Loading Dataset: 2.51s INFO: 2024-10-28 13:16:22,246: llmtf.base.darumeru/RWSD: Processing Dataset: 5.82s INFO: 2024-10-28 13:16:22,246: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: INFO: 2024-10-28 13:16:22,247: llmtf.base.darumeru/RWSD: {'acc': 0.5441176470588235} INFO: 2024-10-28 13:16:22,248: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:16:22,249: llmtf.base.evaluator: mean darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA 0.717 0.780 0.544 0.826 INFO: 2024-10-28 13:16:31,348: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] INFO: 2024-10-28 13:16:31,348: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:16:31,348: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:18:38,554: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 127.21s INFO: 2024-10-28 13:20:06,478: llmtf.base.darumeru/MultiQ: Processing Dataset: 287.18s INFO: 2024-10-28 13:20:06,479: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: INFO: 2024-10-28 13:20:06,480: llmtf.base.darumeru/MultiQ: {'f1': 0.2503859074384594, 'em': 0.14531548757170173} INFO: 2024-10-28 13:20:06,488: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:20:06,489: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RWSD darumeru/ruOpenBookQA 0.587 0.198 0.780 0.544 0.826 INFO: 2024-10-28 13:20:15,334: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] INFO: 2024-10-28 13:20:15,335: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:20:15,335: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:20:18,179: llmtf.base.darumeru/RCB: Loading Dataset: 2.84s INFO: 2024-10-28 13:20:23,505: llmtf.base.darumeru/RCB: Processing Dataset: 5.33s INFO: 2024-10-28 13:20:23,506: llmtf.base.darumeru/RCB: Results for darumeru/RCB: INFO: 2024-10-28 13:20:23,510: llmtf.base.darumeru/RCB: {'acc': 0.5863636363636363, 'f1_macro': 0.5032640286161413} INFO: 2024-10-28 13:20:23,511: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:20:23,512: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA 0.579 0.198 0.780 0.545 0.544 0.826 INFO: 2024-10-28 13:20:32,046: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] INFO: 2024-10-28 13:20:32,046: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:20:32,046: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:20:34,403: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.36s INFO: 2024-10-28 13:20:36,969: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.57s INFO: 2024-10-28 13:20:36,969: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: INFO: 2024-10-28 13:20:36,972: llmtf.base.darumeru/ruWorldTree: {'acc': 0.9047619047619048, 'f1_macro': 0.9038817229146561} INFO: 2024-10-28 13:20:36,972: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:20:36,972: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree 0.633 0.198 0.780 0.545 0.544 0.826 0.904 INFO: 2024-10-28 13:20:45,488: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] INFO: 2024-10-28 13:20:45,488: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:20:45,488: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:20:59,443: llmtf.base.daru/treewayextractive: Loading Dataset: 13.95s INFO: 2024-10-28 13:23:49,533: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 310.98s INFO: 2024-10-28 13:23:49,533: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: INFO: 2024-10-28 13:23:49,597: llmtf.base.nlpcoreteam/ruMMLU: metric subject abstract_algebra 0.480000 anatomy 0.562963 astronomy 0.769737 business_ethics 0.640000 clinical_knowledge 0.671698 college_biology 0.673611 college_chemistry 0.470000 college_computer_science 0.670000 college_mathematics 0.440000 college_medicine 0.589595 college_physics 0.480392 computer_security 0.720000 conceptual_physics 0.655319 econometrics 0.482456 electrical_engineering 0.606897 elementary_mathematics 0.616402 formal_logic 0.428571 global_facts 0.370000 high_school_biology 0.809677 high_school_chemistry 0.571429 high_school_computer_science 0.770000 high_school_european_history 0.751515 high_school_geography 0.782828 high_school_government_and_politics 0.725389 high_school_macroeconomics 0.658974 high_school_mathematics 0.525926 high_school_microeconomics 0.705882 high_school_physics 0.463576 high_school_psychology 0.796330 high_school_statistics 0.606481 high_school_us_history 0.779412 high_school_world_history 0.801688 human_aging 0.632287 human_sexuality 0.717557 international_law 0.743802 jurisprudence 0.675926 logical_fallacies 0.662577 machine_learning 0.482143 management 0.747573 marketing 0.816239 medical_genetics 0.650000 miscellaneous 0.711367 moral_disputes 0.627168 moral_scenarios 0.244693 nutrition 0.689542 philosophy 0.646302 prehistory 0.660494 professional_accounting 0.439716 professional_law 0.411343 professional_medicine 0.613971 professional_psychology 0.591503 public_relations 0.545455 security_studies 0.665306 sociology 0.736318 us_foreign_policy 0.800000 virology 0.500000 world_religions 0.760234 INFO: 2024-10-28 13:23:49,606: llmtf.base.nlpcoreteam/ruMMLU: metric subject STEM 0.600644 humanities 0.630286 other (business, health, misc.) 0.616782 social sciences 0.684000 INFO: 2024-10-28 13:23:49,611: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.6329281396317665} INFO: 2024-10-28 13:23:49,646: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:23:49,648: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.633 0.198 0.780 0.545 0.544 0.826 0.904 0.633 INFO: 2024-10-28 13:23:57,887: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] INFO: 2024-10-28 13:23:57,887: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:23:57,887: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:24:02,221: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.33s INFO: 2024-10-28 13:26:15,188: llmtf.base.daru/treewayextractive: Processing Dataset: 315.74s INFO: 2024-10-28 13:26:15,188: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: INFO: 2024-10-28 13:26:15,447: llmtf.base.daru/treewayextractive: {'r-prec': 0.40380281385281386} INFO: 2024-10-28 13:26:15,501: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:26:15,503: llmtf.base.evaluator: mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.604 0.404 0.198 0.780 0.545 0.544 0.826 0.904 0.633 INFO: 2024-10-28 13:26:24,206: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] INFO: 2024-10-28 13:26:24,207: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:26:24,207: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:26:48,154: llmtf.base.daru/treewayabstractive: Processing Dataset: 165.93s INFO: 2024-10-28 13:26:48,154: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: INFO: 2024-10-28 13:26:48,155: llmtf.base.daru/treewayabstractive: {'rouge1': 0.3489002151166006, 'rouge2': 0.12404569962254197} INFO: 2024-10-28 13:26:48,156: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:26:48,157: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.563 0.236 0.404 0.198 0.780 0.545 0.544 0.826 0.904 0.633 INFO: 2024-10-28 13:28:23,832: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 119.62s INFO: 2024-10-28 13:33:05,781: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 281.95s INFO: 2024-10-28 13:33:05,781: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: INFO: 2024-10-28 13:33:05,844: llmtf.base.nlpcoreteam/enMMLU: metric subject abstract_algebra 0.450000 anatomy 0.725926 astronomy 0.861842 business_ethics 0.750000 clinical_knowledge 0.762264 college_biology 0.854167 college_chemistry 0.510000 college_computer_science 0.720000 college_mathematics 0.470000 college_medicine 0.699422 college_physics 0.509804 computer_security 0.770000 conceptual_physics 0.706383 econometrics 0.605263 electrical_engineering 0.696552 elementary_mathematics 0.666667 formal_logic 0.492063 global_facts 0.420000 high_school_biology 0.861290 high_school_chemistry 0.620690 high_school_computer_science 0.840000 high_school_european_history 0.824242 high_school_geography 0.873737 high_school_government_and_politics 0.927461 high_school_macroeconomics 0.761538 high_school_mathematics 0.566667 high_school_microeconomics 0.873950 high_school_physics 0.582781 high_school_psychology 0.888073 high_school_statistics 0.708333 high_school_us_history 0.838235 high_school_world_history 0.860759 human_aging 0.762332 human_sexuality 0.786260 international_law 0.809917 jurisprudence 0.796296 logical_fallacies 0.828221 machine_learning 0.526786 management 0.854369 marketing 0.914530 medical_genetics 0.810000 miscellaneous 0.848020 moral_disputes 0.736994 moral_scenarios 0.459218 nutrition 0.797386 philosophy 0.723473 prehistory 0.805556 professional_accounting 0.556738 professional_law 0.507823 professional_medicine 0.742647 professional_psychology 0.750000 public_relations 0.636364 security_studies 0.759184 sociology 0.845771 us_foreign_policy 0.850000 virology 0.506024 world_religions 0.853801 INFO: 2024-10-28 13:33:05,852: llmtf.base.nlpcoreteam/enMMLU: metric subject STEM 0.662331 humanities 0.733585 other (business, health, misc.) 0.724976 social sciences 0.796467 INFO: 2024-10-28 13:33:05,857: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.7293395108036221} INFO: 2024-10-28 13:33:05,908: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:33:05,910: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.580 0.236 0.404 0.198 0.780 0.545 0.544 0.826 0.904 0.729 0.633 INFO: 2024-10-28 13:33:14,562: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] INFO: 2024-10-28 13:33:14,562: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-28 13:33:14,562: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-28 13:33:17,057: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.49s INFO: 2024-10-28 13:35:21,669: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 124.61s INFO: 2024-10-28 13:35:21,670: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: INFO: 2024-10-28 13:35:21,670: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.9953318595732386, 'len': 0.9990656928305265, 'lcs': 1.0} INFO: 2024-10-28 13:35:21,671: llmtf.base.evaluator: Ended eval INFO: 2024-10-28 13:35:21,672: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.618 0.236 0.404 0.198 0.780 0.545 0.544 1.000 0.826 0.904 0.729 0.633