giulio98
/

codegen-350M-multi-xlcost

Text Generation

Inference Endpoints

Model card Files Files and versions Community

giulio98 commited on Nov 9, 2022

Commit

7c997a4

•

1 Parent(s): e9899e6

Update README.md

Files changed (1) hide show

README.md +4 -4

README.md CHANGED Viewed

@@ -78,10 +78,10 @@ The training was executed on 1 x V100 (16GB) GPU for 6h 42m
 We evaluated the model on the first 400 samples of XLCOST's [XLCost-single-prompt test split](https://huggingface.co/datasets/giulio98/xlcost-single-prompt/viewer/Python/test) and comparing the outputs of the generated codes with respect to the expected output using pass@k metric.
-| Metric | codegen-350M-multi-xlcost |
-|--------|-----|
-|pass@1 | 3.70% |
-|pass@10 | 14.5% |
 The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests.

 We evaluated the model on the first 400 samples of XLCOST's [XLCost-single-prompt test split](https://huggingface.co/datasets/giulio98/xlcost-single-prompt/viewer/Python/test) and comparing the outputs of the generated codes with respect to the expected output using pass@k metric.
+| Metric | codegen-350M-multi-xlcost | codegen-350M-mono(zero-shot) | codegen-350M-mono (one-shot) | codegen-350M-mono(few-shot)
+|--------|-----|-----|-----|-----|
+|pass@1 | 3.70% | 0.4% | 0.35% | 0.48% |
+|pass@10 | 14.5% | 3.5% | 3 % | 3.75% |
 The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests.