INFO: 2024-10-17 21:30:14,019: llmtf.base.evaluator: Starting eval on ['darumeru/multiq'] INFO: 2024-10-17 21:30:14,019: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:30:14,019: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:30:20,481: llmtf.base.darumeru/MultiQ: Loading Dataset: 6.46s INFO: 2024-10-17 21:35:59,593: llmtf.base.darumeru/MultiQ: Processing Dataset: 339.11s INFO: 2024-10-17 21:35:59,593: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ: INFO: 2024-10-17 21:35:59,594: llmtf.base.darumeru/MultiQ: {'f1': 0.3346248767848689, 'em': 0.22275334608030592} INFO: 2024-10-17 21:35:59,599: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:35:59,599: llmtf.base.evaluator: mean darumeru/MultiQ 0.279 0.279 INFO: 2024-10-17 21:36:08,809: llmtf.base.evaluator: Starting eval on ['darumeru/parus'] INFO: 2024-10-17 21:36:08,810: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:36:08,810: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:36:12,969: llmtf.base.darumeru/PARus: Loading Dataset: 4.16s INFO: 2024-10-17 21:36:18,316: llmtf.base.darumeru/PARus: Processing Dataset: 5.35s INFO: 2024-10-17 21:36:18,317: llmtf.base.darumeru/PARus: Results for darumeru/PARus: INFO: 2024-10-17 21:36:18,327: llmtf.base.darumeru/PARus: {'acc': 0.7} INFO: 2024-10-17 21:36:18,327: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:36:18,328: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus 0.489 0.279 0.700 INFO: 2024-10-17 21:36:27,550: llmtf.base.evaluator: Starting eval on ['darumeru/rcb'] INFO: 2024-10-17 21:36:27,550: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:36:27,551: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:36:31,450: llmtf.base.darumeru/RCB: Loading Dataset: 3.90s INFO: 2024-10-17 21:36:38,683: llmtf.base.darumeru/RCB: Processing Dataset: 7.23s INFO: 2024-10-17 21:36:38,683: llmtf.base.darumeru/RCB: Results for darumeru/RCB: INFO: 2024-10-17 21:36:38,686: llmtf.base.darumeru/RCB: {'acc': 0.5454545454545454, 'f1_macro': 0.49090309951702227} INFO: 2024-10-17 21:36:38,687: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:36:38,688: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB 0.499 0.279 0.700 0.518 INFO: 2024-10-17 21:36:48,734: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa'] INFO: 2024-10-17 21:36:48,735: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:36:48,735: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:36:54,900: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 6.17s INFO: 2024-10-17 21:38:00,519: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 65.62s INFO: 2024-10-17 21:38:00,520: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA: INFO: 2024-10-17 21:38:00,532: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.7302405498281787, 'f1_macro': 0.7304546157096631} INFO: 2024-10-17 21:38:00,541: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:38:00,542: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/ruOpenBookQA 0.557 0.279 0.700 0.518 0.730 INFO: 2024-10-17 21:38:09,745: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree'] INFO: 2024-10-17 21:38:09,745: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:38:09,745: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:38:14,102: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 4.36s INFO: 2024-10-17 21:38:16,932: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.83s INFO: 2024-10-17 21:38:16,933: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree: INFO: 2024-10-17 21:38:16,936: llmtf.base.darumeru/ruWorldTree: {'acc': 0.9047619047619048, 'f1_macro': 0.9043404138496471} INFO: 2024-10-17 21:38:16,936: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:38:16,937: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/ruOpenBookQA darumeru/ruWorldTree 0.626 0.279 0.700 0.518 0.730 0.905 INFO: 2024-10-17 21:38:26,077: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd'] INFO: 2024-10-17 21:38:26,077: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:38:26,077: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:38:30,781: llmtf.base.darumeru/RWSD: Loading Dataset: 4.70s INFO: 2024-10-17 21:38:36,497: llmtf.base.darumeru/RWSD: Processing Dataset: 5.72s INFO: 2024-10-17 21:38:36,497: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD: INFO: 2024-10-17 21:38:36,498: llmtf.base.darumeru/RWSD: {'acc': 0.6029411764705882} INFO: 2024-10-17 21:38:36,499: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:38:36,500: llmtf.base.evaluator: mean darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree 0.622 0.279 0.700 0.518 0.603 0.730 0.905 INFO: 2024-10-17 21:38:45,688: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive'] INFO: 2024-10-17 21:38:45,688: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:38:45,688: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:39:02,002: llmtf.base.daru/treewayextractive: Loading Dataset: 16.31s INFO: 2024-10-17 21:42:05,777: llmtf.base.daru/treewayextractive: Processing Dataset: 183.77s INFO: 2024-10-17 21:42:05,777: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive: INFO: 2024-10-17 21:42:06,010: llmtf.base.daru/treewayextractive: {'r-prec': 0.3917218614718615} INFO: 2024-10-17 21:42:06,052: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:42:06,054: llmtf.base.evaluator: mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree 0.589 0.392 0.279 0.700 0.518 0.603 0.730 0.905 INFO: 2024-10-17 21:42:15,170: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu'] INFO: 2024-10-17 21:42:15,170: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:42:15,170: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:46:47,282: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 272.11s INFO: 2024-10-17 21:56:29,398: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 582.12s INFO: 2024-10-17 21:56:29,399: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU: INFO: 2024-10-17 21:56:29,464: llmtf.base.nlpcoreteam/ruMMLU: metric subject abstract_algebra 0.340000 anatomy 0.414815 astronomy 0.611842 business_ethics 0.610000 clinical_knowledge 0.554717 college_biology 0.548611 college_chemistry 0.380000 college_computer_science 0.450000 college_mathematics 0.400000 college_medicine 0.526012 college_physics 0.470588 computer_security 0.620000 conceptual_physics 0.565957 econometrics 0.377193 electrical_engineering 0.537931 elementary_mathematics 0.529101 formal_logic 0.365079 global_facts 0.360000 high_school_biology 0.664516 high_school_chemistry 0.487685 high_school_computer_science 0.700000 high_school_european_history 0.751515 high_school_geography 0.722222 high_school_government_and_politics 0.564767 high_school_macroeconomics 0.528205 high_school_mathematics 0.433333 high_school_microeconomics 0.533613 high_school_physics 0.403974 high_school_psychology 0.713761 high_school_statistics 0.523148 high_school_us_history 0.661765 high_school_world_history 0.717300 human_aging 0.587444 human_sexuality 0.618321 international_law 0.735537 jurisprudence 0.666667 logical_fallacies 0.564417 machine_learning 0.392857 management 0.650485 marketing 0.752137 medical_genetics 0.580000 miscellaneous 0.632184 moral_disputes 0.583815 moral_scenarios 0.299441 nutrition 0.637255 philosophy 0.617363 prehistory 0.561728 professional_accounting 0.386525 professional_law 0.377445 professional_medicine 0.481618 professional_psychology 0.516340 public_relations 0.500000 security_studies 0.648980 sociology 0.756219 us_foreign_policy 0.720000 virology 0.439759 world_religions 0.719298 INFO: 2024-10-17 21:56:29,473: llmtf.base.nlpcoreteam/ruMMLU: metric subject STEM 0.503308 humanities 0.586259 other (business, health, misc.) 0.543782 social sciences 0.599968 INFO: 2024-10-17 21:56:29,478: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.5583294528508019} INFO: 2024-10-17 21:56:29,516: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 21:56:29,518: llmtf.base.evaluator: mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/ruMMLU 0.586 0.392 0.279 0.700 0.518 0.603 0.730 0.905 0.558 INFO: 2024-10-17 21:56:39,535: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu'] INFO: 2024-10-17 21:56:39,536: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 21:56:39,536: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 21:58:54,966: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 135.43s INFO: 2024-10-17 22:08:04,419: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 549.45s INFO: 2024-10-17 22:08:04,426: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU: INFO: 2024-10-17 22:08:04,492: llmtf.base.nlpcoreteam/enMMLU: metric subject abstract_algebra 0.380000 anatomy 0.637037 astronomy 0.717105 business_ethics 0.700000 clinical_knowledge 0.705660 college_biology 0.715278 college_chemistry 0.470000 college_computer_science 0.580000 college_mathematics 0.330000 college_medicine 0.664740 college_physics 0.509804 computer_security 0.740000 conceptual_physics 0.642553 econometrics 0.508772 electrical_engineering 0.600000 elementary_mathematics 0.547619 formal_logic 0.412698 global_facts 0.360000 high_school_biology 0.783871 high_school_chemistry 0.581281 high_school_computer_science 0.710000 high_school_european_history 0.800000 high_school_geography 0.757576 high_school_government_and_politics 0.854922 high_school_macroeconomics 0.679487 high_school_mathematics 0.455556 high_school_microeconomics 0.773109 high_school_physics 0.437086 high_school_psychology 0.844037 high_school_statistics 0.652778 high_school_us_history 0.833333 high_school_world_history 0.843882 human_aging 0.677130 human_sexuality 0.786260 international_law 0.768595 jurisprudence 0.814815 logical_fallacies 0.803681 machine_learning 0.446429 management 0.786408 marketing 0.858974 medical_genetics 0.760000 miscellaneous 0.795658 moral_disputes 0.667630 moral_scenarios 0.311732 nutrition 0.732026 philosophy 0.704180 prehistory 0.712963 professional_accounting 0.503546 professional_law 0.457627 professional_medicine 0.658088 professional_psychology 0.668301 public_relations 0.709091 security_studies 0.697959 sociology 0.800995 us_foreign_policy 0.800000 virology 0.506024 world_religions 0.801170 INFO: 2024-10-17 22:08:04,506: llmtf.base.nlpcoreteam/enMMLU: metric subject STEM 0.572187 humanities 0.687100 other (business, health, misc.) 0.667521 social sciences 0.740042 INFO: 2024-10-17 22:08:04,511: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.6667125709237595} INFO: 2024-10-17 22:08:04,554: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 22:08:04,556: llmtf.base.evaluator: mean daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.595 0.392 0.279 0.700 0.518 0.603 0.730 0.905 0.667 0.558 INFO: 2024-10-17 22:08:14,512: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive'] INFO: 2024-10-17 22:08:14,513: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 22:08:14,513: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 22:08:18,791: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.28s INFO: 2024-10-17 22:11:46,260: llmtf.base.daru/treewayabstractive: Processing Dataset: 207.47s INFO: 2024-10-17 22:11:46,260: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive: INFO: 2024-10-17 22:11:46,261: llmtf.base.daru/treewayabstractive: {'rouge1': 0.33109987599556284, 'rouge2': 0.11202889150257295} INFO: 2024-10-17 22:11:46,262: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 22:11:46,263: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.557 0.222 0.392 0.279 0.700 0.518 0.603 0.730 0.905 0.667 0.558 INFO: 2024-10-17 22:11:55,717: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru'] INFO: 2024-10-17 22:11:55,717: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077] INFO: 2024-10-17 22:11:55,717: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>'] INFO: 2024-10-17 22:11:59,846: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 4.13s INFO: 2024-10-17 22:14:29,975: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 150.13s INFO: 2024-10-17 22:14:29,975: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru: INFO: 2024-10-17 22:14:29,976: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.993754090002875, 'len': 0.9986883734384026, 'lcs': 0.98} INFO: 2024-10-17 22:14:29,977: llmtf.base.evaluator: Ended eval INFO: 2024-10-17 22:14:29,977: llmtf.base.evaluator: mean daru/treewayabstractive daru/treewayextractive darumeru/MultiQ darumeru/PARus darumeru/RCB darumeru/RWSD darumeru/cp_para_ru darumeru/ruOpenBookQA darumeru/ruWorldTree nlpcoreteam/enMMLU nlpcoreteam/ruMMLU 0.596 0.222 0.392 0.279 0.700 0.518 0.603 0.980 0.730 0.905 0.667 0.558