UNA-34BeagleSimpleMath-32K-v1

This is a fine-tuned version of fblgit/UNA-34Beagles-32K-v1 using fblgit/simple-math dataset. Powered by The Bagel v0.3 and Yi-34B

Trained on AXOLOTL!

34BEAGLES MATH EVALS

|Tasks|Version|  Filter  |n-shot|  Metric   |Value |   |Stderr|
|-----|-------|----------|-----:|-----------|-----:|---|-----:|
|gsm8k|Yaml   |get-answer|     5|exact_match|0.6505|±  |0.0131|

|    Tasks     |Version|Filter|n-shot| Metric |Value |   |Stderr|
|--------------|-------|------|-----:|--------|-----:|---|-----:|
|arc_challenge |Yaml   |none  |    25|acc     |0.7090|±  |0.0133|
|              |       |none  |    25|acc_norm|0.7329|±  |0.0129|
|truthfulqa_mc2|Yaml   |none  |     0|acc     |0.7378|±  |0.0141|

|                 Tasks                 |Version|Filter|n-shot|Metric|Value |   |Stderr|
|---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu                                   |N/A    |none  |     0|acc   |0.7524|±  |0.1045|
| - humanities                          |N/A    |none  |     0|acc   |0.7307|±  |0.0846|
|  - formal_logic                       |Yaml   |none  |     0|acc   |0.5873|±  |0.0440|
|  - high_school_european_history       |Yaml   |none  |     0|acc   |0.8667|±  |0.0265|
|  - high_school_us_history             |Yaml   |none  |     0|acc   |0.9167|±  |0.0194|
|  - high_school_world_history          |Yaml   |none  |     0|acc   |0.9114|±  |0.0185|
|  - international_law                  |Yaml   |none  |     0|acc   |0.8430|±  |0.0332|
|  - jurisprudence                      |Yaml   |none  |     0|acc   |0.8704|±  |0.0325|
|  - logical_fallacies                  |Yaml   |none  |     0|acc   |0.8589|±  |0.0274|
|  - moral_disputes                     |Yaml   |none  |     0|acc   |0.7717|±  |0.0226|
|  - moral_scenarios                    |Yaml   |none  |     0|acc   |0.7374|±  |0.0147|
|  - philosophy                         |Yaml   |none  |     0|acc   |0.8006|±  |0.0227|
|  - prehistory                         |Yaml   |none  |     0|acc   |0.8549|±  |0.0196|
|  - professional_law                   |Yaml   |none  |     0|acc   |0.5724|±  |0.0126|
|  - world_religions                    |Yaml   |none  |     0|acc   |0.8830|±  |0.0246|
| - other                               |N/A    |none  |     0|acc   |0.7937|±  |0.1029|
|  - business_ethics                    |Yaml   |none  |     0|acc   |0.7800|±  |0.0416|
|  - clinical_knowledge                 |Yaml   |none  |     0|acc   |0.8000|±  |0.0246|
|  - college_medicine                   |Yaml   |none  |     0|acc   |0.6936|±  |0.0351|
|  - global_facts                       |Yaml   |none  |     0|acc   |0.5500|±  |0.0500|
|  - human_aging                        |Yaml   |none  |     0|acc   |0.7534|±  |0.0289|
|  - management                         |Yaml   |none  |     0|acc   |0.8447|±  |0.0359|
|  - marketing                          |Yaml   |none  |     0|acc   |0.9316|±  |0.0165|
|  - medical_genetics                   |Yaml   |none  |     0|acc   |0.8700|±  |0.0338|
|  - miscellaneous                      |Yaml   |none  |     0|acc   |0.8953|±  |0.0109|
|  - nutrition                          |Yaml   |none  |     0|acc   |0.8170|±  |0.0221|
|  - professional_accounting            |Yaml   |none  |     0|acc   |0.6277|±  |0.0288|
|  - professional_medicine              |Yaml   |none  |     0|acc   |0.8015|±  |0.0242|
|  - virology                           |Yaml   |none  |     0|acc   |0.5723|±  |0.0385|
| - social_sciences                     |N/A    |none  |     0|acc   |0.8274|±  |0.0667|
|  - econometrics                       |Yaml   |none  |     0|acc   |0.6140|±  |0.0458|
|  - high_school_geography              |Yaml   |none  |     0|acc   |0.8889|±  |0.0224|
|  - high_school_government_and_politics|Yaml   |none  |     0|acc   |0.9482|±  |0.0160|
|  - high_school_macroeconomics         |Yaml   |none  |     0|acc   |0.7897|±  |0.0207|
|  - high_school_microeconomics         |Yaml   |none  |     0|acc   |0.8697|±  |0.0219|
|  - high_school_psychology             |Yaml   |none  |     0|acc   |0.8899|±  |0.0134|
|  - human_sexuality                    |Yaml   |none  |     0|acc   |0.8550|±  |0.0309|
|  - professional_psychology            |Yaml   |none  |     0|acc   |0.7745|±  |0.0169|
|  - public_relations                   |Yaml   |none  |     0|acc   |0.7000|±  |0.0439|
|  - security_studies                   |Yaml   |none  |     0|acc   |0.7796|±  |0.0265|
|  - sociology                          |Yaml   |none  |     0|acc   |0.8657|±  |0.0241|
|  - us_foreign_policy                  |Yaml   |none  |     0|acc   |0.8900|±  |0.0314|
| - stem                                |N/A    |none  |     0|acc   |0.6708|±  |0.1236|
|  - abstract_algebra                   |Yaml   |none  |     0|acc   |0.4900|±  |0.0502|
|  - anatomy                            |Yaml   |none  |     0|acc   |0.7259|±  |0.0385|
|  - astronomy                          |Yaml   |none  |     0|acc   |0.8487|±  |0.0292|
|  - college_biology                    |Yaml   |none  |     0|acc   |0.8750|±  |0.0277|
|  - college_chemistry                  |Yaml   |none  |     0|acc   |0.5200|±  |0.0502|
|  - college_computer_science           |Yaml   |none  |     0|acc   |0.6200|±  |0.0488|
|  - college_mathematics                |Yaml   |none  |     0|acc   |0.4300|±  |0.0498|
|  - college_physics                    |Yaml   |none  |     0|acc   |0.5686|±  |0.0493|
|  - computer_security                  |Yaml   |none  |     0|acc   |0.7800|±  |0.0416|
|  - conceptual_physics                 |Yaml   |none  |     0|acc   |0.7404|±  |0.0287|
|  - electrical_engineering             |Yaml   |none  |     0|acc   |0.7172|±  |0.0375|
|  - elementary_mathematics             |Yaml   |none  |     0|acc   |0.6720|±  |0.0242|
|  - high_school_biology                |Yaml   |none  |     0|acc   |0.9032|±  |0.0168|
|  - high_school_chemistry              |Yaml   |none  |     0|acc   |0.6256|±  |0.0341|
|  - high_school_computer_science       |Yaml   |none  |     0|acc   |0.7800|±  |0.0416|
|  - high_school_mathematics            |Yaml   |none  |     0|acc   |0.4667|±  |0.0304|
|  - high_school_physics                |Yaml   |none  |     0|acc   |0.5033|±  |0.0408|
|  - high_school_statistics             |Yaml   |none  |     0|acc   |0.6435|±  |0.0327|
|  - machine_learning                   |Yaml   |none  |     0|acc   |0.5536|±  |0.0472|

|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.7524|±  |0.1045|
| - humanities     |N/A    |none  |     0|acc   |0.7307|±  |0.0846|
| - other          |N/A    |none  |     0|acc   |0.7937|±  |0.1029|
| - social_sciences|N/A    |none  |     0|acc   |0.8274|±  |0.0667|
| - stem           |N/A    |none  |     0|acc   |0.6708|±  |0.1236|

34BEAGLES (The Base Model)

|    Tasks     |Version|Filter|n-shot| Metric |Value |   |Stderr|
|--------------|-------|------|-----:|--------|-----:|---|-----:|
|arc_challenge |Yaml   |none  |    25|acc     |0.7039|±  |0.0133|
|              |       |none  |    25|acc_norm|0.7321|±  |0.0129|
|truthfulqa_mc2|Yaml   |none  |     0|acc     |0.7387|±  |0.0141|

|Tasks|Version|  Filter  |n-shot|  Metric   |Value |   |Stderr|
|-----|-------|----------|-----:|-----------|-----:|---|-----:|
|gsm8k|Yaml   |get-answer|     5|exact_match|0.6399|±  |0.0132|

|                 Tasks                 |Version|Filter|n-shot|Metric|Value |   |Stderr|
|---------------------------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu                                   |N/A    |none  |     0|acc   |0.7477|±  |0.1079|
| - humanities                          |N/A    |none  |     0|acc   |0.7188|±  |0.0855|
|  - formal_logic                       |Yaml   |none  |     0|acc   |0.5794|±  |0.0442|
|  - high_school_european_history       |Yaml   |none  |     0|acc   |0.8667|±  |0.0265|
|  - high_school_us_history             |Yaml   |none  |     0|acc   |0.9069|±  |0.0204|
|  - high_school_world_history          |Yaml   |none  |     0|acc   |0.9072|±  |0.0189|
|  - international_law                  |Yaml   |none  |     0|acc   |0.8264|±  |0.0346|
|  - jurisprudence                      |Yaml   |none  |     0|acc   |0.8796|±  |0.0315|
|  - logical_fallacies                  |Yaml   |none  |     0|acc   |0.8405|±  |0.0288|
|  - moral_disputes                     |Yaml   |none  |     0|acc   |0.7746|±  |0.0225|
|  - moral_scenarios                    |Yaml   |none  |     0|acc   |0.6972|±  |0.0154|
|  - philosophy                         |Yaml   |none  |     0|acc   |0.8006|±  |0.0227|
|  - prehistory                         |Yaml   |none  |     0|acc   |0.8580|±  |0.0194|
|  - professional_law                   |Yaml   |none  |     0|acc   |0.5645|±  |0.0127|
|  - world_religions                    |Yaml   |none  |     0|acc   |0.8713|±  |0.0257|
| - other                               |N/A    |none  |     0|acc   |0.7950|±  |0.1057|
|  - business_ethics                    |Yaml   |none  |     0|acc   |0.7700|±  |0.0423|
|  - clinical_knowledge                 |Yaml   |none  |     0|acc   |0.8038|±  |0.0244|
|  - college_medicine                   |Yaml   |none  |     0|acc   |0.7110|±  |0.0346|
|  - global_facts                       |Yaml   |none  |     0|acc   |0.5500|±  |0.0500|
|  - human_aging                        |Yaml   |none  |     0|acc   |0.7265|±  |0.0299|
|  - management                         |Yaml   |none  |     0|acc   |0.8544|±  |0.0349|
|  - marketing                          |Yaml   |none  |     0|acc   |0.9444|±  |0.0150|
|  - medical_genetics                   |Yaml   |none  |     0|acc   |0.8800|±  |0.0327|
|  - miscellaneous                      |Yaml   |none  |     0|acc   |0.8978|±  |0.0108|
|  - nutrition                          |Yaml   |none  |     0|acc   |0.8170|±  |0.0221|
|  - professional_accounting            |Yaml   |none  |     0|acc   |0.6312|±  |0.0288|
|  - professional_medicine              |Yaml   |none  |     0|acc   |0.8051|±  |0.0241|
|  - virology                           |Yaml   |none  |     0|acc   |0.5602|±  |0.0386|
| - social_sciences                     |N/A    |none  |     0|acc   |0.8297|±  |0.0664|
|  - econometrics                       |Yaml   |none  |     0|acc   |0.6140|±  |0.0458|
|  - high_school_geography              |Yaml   |none  |     0|acc   |0.8939|±  |0.0219|
|  - high_school_government_and_politics|Yaml   |none  |     0|acc   |0.9482|±  |0.0160|
|  - high_school_macroeconomics         |Yaml   |none  |     0|acc   |0.7974|±  |0.0204|
|  - high_school_microeconomics         |Yaml   |none  |     0|acc   |0.8655|±  |0.0222|
|  - high_school_psychology             |Yaml   |none  |     0|acc   |0.8936|±  |0.0132|
|  - human_sexuality                    |Yaml   |none  |     0|acc   |0.8473|±  |0.0315|
|  - professional_psychology            |Yaml   |none  |     0|acc   |0.7778|±  |0.0168|
|  - public_relations                   |Yaml   |none  |     0|acc   |0.7000|±  |0.0439|
|  - security_studies                   |Yaml   |none  |     0|acc   |0.7837|±  |0.0264|
|  - sociology                          |Yaml   |none  |     0|acc   |0.8657|±  |0.0241|
|  - us_foreign_policy                  |Yaml   |none  |     0|acc   |0.8900|±  |0.0314|
| - stem                                |N/A    |none  |     0|acc   |0.6641|±  |0.1291|
|  - abstract_algebra                   |Yaml   |none  |     0|acc   |0.4800|±  |0.0502|
|  - anatomy                            |Yaml   |none  |     0|acc   |0.7407|±  |0.0379|
|  - astronomy                          |Yaml   |none  |     0|acc   |0.8618|±  |0.0281|
|  - college_biology                    |Yaml   |none  |     0|acc   |0.8611|±  |0.0289|
|  - college_chemistry                  |Yaml   |none  |     0|acc   |0.5300|±  |0.0502|
|  - college_computer_science           |Yaml   |none  |     0|acc   |0.6100|±  |0.0490|
|  - college_mathematics                |Yaml   |none  |     0|acc   |0.3800|±  |0.0488|
|  - college_physics                    |Yaml   |none  |     0|acc   |0.5588|±  |0.0494|
|  - computer_security                  |Yaml   |none  |     0|acc   |0.8000|±  |0.0402|
|  - conceptual_physics                 |Yaml   |none  |     0|acc   |0.7319|±  |0.0290|
|  - electrical_engineering             |Yaml   |none  |     0|acc   |0.7034|±  |0.0381|
|  - elementary_mathematics             |Yaml   |none  |     0|acc   |0.6587|±  |0.0244|
|  - high_school_biology                |Yaml   |none  |     0|acc   |0.8935|±  |0.0175|
|  - high_school_chemistry              |Yaml   |none  |     0|acc   |0.6305|±  |0.0340|
|  - high_school_computer_science       |Yaml   |none  |     0|acc   |0.7700|±  |0.0423|
|  - high_school_mathematics            |Yaml   |none  |     0|acc   |0.4296|±  |0.0302|
|  - high_school_physics                |Yaml   |none  |     0|acc   |0.5166|±  |0.0408|
|  - high_school_statistics             |Yaml   |none  |     0|acc   |0.6528|±  |0.0325|
|  - machine_learning                   |Yaml   |none  |     0|acc   |0.5536|±  |0.0472|

|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.7477|±  |0.1079|
| - humanities     |N/A    |none  |     0|acc   |0.7188|±  |0.0855|
| - other          |N/A    |none  |     0|acc   |0.7950|±  |0.1057|
| - social_sciences|N/A    |none  |     0|acc   |0.8297|±  |0.0664|
| - stem           |N/A    |none  |     0|acc   |0.6641|±  |0.1291|

So I guess, SimpleMath: 2+2=4 4-1=3 SIMPLE-MATH .. works! :)