switch Base vs Instruct
Browse files
README.md
CHANGED
|
@@ -319,14 +319,14 @@ Fairness Evaluations:
|
|
| 319 |
IDEFICS vs IDEFICS-instruct.
|
| 320 |
| Model | Shots | <nobr>VQAv2<br>OE VQA acc.</nobr> | <nobr>OKVQA<br>OE VQA acc.</nobr> | <nobr>TextVQA<br>OE VQA acc.</nobr> | <nobr>VizWiz<br>OE VQA acc.</nobr> | <nobr>TextCaps<br>CIDEr</nobr> | <nobr>Coco<br>CIDEr</nobr> | <nobr>NoCaps<br>CIDEr</nobr> | <nobr>Flickr<br>CIDEr</nobr> | <nobr>VisDial<br>NDCG</nobr> | <nobr>HatefulMemes<br>ROC AUC</nobr> | <nobr>ScienceQA<br>acc.</nobr> | <nobr>RenderedSST2<br>acc.</nobr> | <nobr>Winoground<br>group (text/image)</nobr> |
|
| 321 |
|:----------------------------------------|:--------|---------------------:|---------------------:|-----------------------:|----------------------:|-------------------:|---------------:|-----------------:|-----------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------:|
|
| 322 |
-
| Difference IDEFICS 80B
|
| 323 |
| | 4 | 4.0 | 1.7 | 3.5 | -0.7 | - | 6.6 | 4.4 | -6.6 | 0.5 | -0.3 | 1.6 | -1.1 | - |
|
| 324 |
| | 8 | 3.4 | 1.8 | 2.5 | -1.3 | -4.9 | 2.5 | -0.9 | -5.9 | 0.3 | -0.2 | - | 0.8 | - |
|
| 325 |
| | 16 | 3.2 | 1.4 | 2.8 | 0.4 | -4.5 | 4.0 | 0.4 | -4.1 | - | 0.7 | - | 2.4 | - |
|
| 326 |
| | 32 | 2.9 | 1.8 | 2.6 | 1.2 | -3.0 | 6.5 | 1.0 | -2.7 | - | 2.4 | - | 3.2 | - |
|
| 327 |
| Average Difference 80B | | -1.8 | -0.3 | 2.6 | -2.0 | 1.3 | 9.0 | 8.9 | -1.5 | 0.4 | 0.2 | 1.1 | 2.4 | 1.2 |
|
| 328 |
<br>
|
| 329 |
-
| Difference IDEFICS 9B
|
| 330 |
| | 4 | 10.8 | 3.3 | 3.4 | 2.1 | 8.2 | 35.1 | 19.6 | 15.0 | 1.0 | 1.1 | 16.4 | -1.8 | - |
|
| 331 |
| | 8 | 10.2 | 3.1 | 3.5 | 1.6 | 6.7 | 31.8 | 14.8 | 13.6 | 0.6 | 0.6 | - | -4.9 | - |
|
| 332 |
| | 16 | 9.8 | 3.3 | 3.7 | 2.3 | 2.7 | 29.1 | 12.2 | 11.4 | - | 0.7 | - | -4.6 | - |
|
|
|
|
| 319 |
IDEFICS vs IDEFICS-instruct.
|
| 320 |
| Model | Shots | <nobr>VQAv2<br>OE VQA acc.</nobr> | <nobr>OKVQA<br>OE VQA acc.</nobr> | <nobr>TextVQA<br>OE VQA acc.</nobr> | <nobr>VizWiz<br>OE VQA acc.</nobr> | <nobr>TextCaps<br>CIDEr</nobr> | <nobr>Coco<br>CIDEr</nobr> | <nobr>NoCaps<br>CIDEr</nobr> | <nobr>Flickr<br>CIDEr</nobr> | <nobr>VisDial<br>NDCG</nobr> | <nobr>HatefulMemes<br>ROC AUC</nobr> | <nobr>ScienceQA<br>acc.</nobr> | <nobr>RenderedSST2<br>acc.</nobr> | <nobr>Winoground<br>group (text/image)</nobr> |
|
| 321 |
|:----------------------------------------|:--------|---------------------:|---------------------:|-----------------------:|----------------------:|-------------------:|---------------:|-----------------:|-----------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------:|
|
| 322 |
+
| Difference IDEFICS 80B Instruct vs Base | 0 | -22.7 | -8.2 | 1.9 | -9.8 | 19.7 | 25.4 | 39.5 | 11.7 | 0.4 | -1.7 | 0.5 | 6.8 | 1.2 |
|
| 323 |
| | 4 | 4.0 | 1.7 | 3.5 | -0.7 | - | 6.6 | 4.4 | -6.6 | 0.5 | -0.3 | 1.6 | -1.1 | - |
|
| 324 |
| | 8 | 3.4 | 1.8 | 2.5 | -1.3 | -4.9 | 2.5 | -0.9 | -5.9 | 0.3 | -0.2 | - | 0.8 | - |
|
| 325 |
| | 16 | 3.2 | 1.4 | 2.8 | 0.4 | -4.5 | 4.0 | 0.4 | -4.1 | - | 0.7 | - | 2.4 | - |
|
| 326 |
| | 32 | 2.9 | 1.8 | 2.6 | 1.2 | -3.0 | 6.5 | 1.0 | -2.7 | - | 2.4 | - | 3.2 | - |
|
| 327 |
| Average Difference 80B | | -1.8 | -0.3 | 2.6 | -2.0 | 1.3 | 9.0 | 8.9 | -1.5 | 0.4 | 0.2 | 1.1 | 2.4 | 1.2 |
|
| 328 |
<br>
|
| 329 |
+
| Difference IDEFICS 9B Instruct vs Base | 0 | 15.0 | 7.6 | 3.3 | 5.6 | 41.7 | 83.0 | 64.3 | 44.6 | 0.5 | 1.8 | 16.4 | 1.0 | 0.8 |
|
| 330 |
| | 4 | 10.8 | 3.3 | 3.4 | 2.1 | 8.2 | 35.1 | 19.6 | 15.0 | 1.0 | 1.1 | 16.4 | -1.8 | - |
|
| 331 |
| | 8 | 10.2 | 3.1 | 3.5 | 1.6 | 6.7 | 31.8 | 14.8 | 13.6 | 0.6 | 0.6 | - | -4.9 | - |
|
| 332 |
| | 16 | 9.8 | 3.3 | 3.7 | 2.3 | 2.7 | 29.1 | 12.2 | 11.4 | - | 0.7 | - | -4.6 | - |
|