Add fairness evals for Idefics instruct

README.md

@@ -301,6 +301,20 @@ Idefics Instruct Evaluations:
 |                      |      16 |                 66.8 |                 51.7 |                   31.6 |                  44.8 |               70.2 |          128.8 |            101.5 |             75.8 |             - |                     51.7 |                   - |                      63.3 |                              - |
 |                      |      32 |                 66.9 |                 52.3 |                   32.0 |                  46.0 |               71.7 |          127.8 |            101.0 |             76.3 |             - |                     50.8 |                   - |                      60.9 |                              - |
 
+Fairness Evaluations:
+| Model | Shots | <nobr>FairFaceGender<br>acc.</nobr> | <nobr>FairFaceRace<br>acc.</nobr> | <nobr>FairFaceAge<br>acc.</nobr> |
+|:---------------------|--------:|----------------------------:|--------------------------:|-------------------------:|
+| 80B IDEFICS Instruct |       0 |                        95.7 |                      63.4 |                     47.1 |
+|                      |       4 |                        95.6 |                      51.4 |                     48.3 |
+|                      |       8 |                        95.8 |                      51.0 |                     51.1 |
+|                      |      16 |                        96.1 |                      47.6 |                     51.8 |
+|                      |      32 |                        96.2 |                      36.8 |                     50.3 |
+| 9B IDEFICS Instruct  |       0 |                        92.7 |                      59.6 |                     43.9 |
+|                      |       4 |                        95.2 |                      43.3 |                     38.7 |
+|                      |       8 |                        95.8 |                      51.7 |                     40.1 |
+|                      |      16 |                        96.1 |                      58.9 |                     41.7 |
+|                      |      32 |                        96.1 |                      59.7 |                     44.8 |
+
 IDEFICS vs IDEFICS-instruct.
 | Model | Shots | <nobr>VQAv2<br>OE VQA acc.</nobr> | <nobr>OKVQA<br>OE VQA acc.</nobr> | <nobr>TextVQA<br>OE VQA acc.</nobr> | <nobr>VizWiz<br>OE VQA acc.</nobr> | <nobr>TextCaps<br>CIDEr</nobr> | <nobr>Coco<br>CIDEr</nobr> | <nobr>NoCaps<br>CIDEr</nobr> | <nobr>Flickr<br>CIDEr</nobr> | <nobr>VisDial<br>NDCG</nobr> | <nobr>HatefulMemes<br>ROC AUC</nobr> | <nobr>ScienceQA<br>acc.</nobr> | <nobr>RenderedSST2<br>acc.</nobr> | <nobr>Winoground<br>group (text/image)</nobr> |
 |:----------------------------------------|:--------|---------------------:|---------------------:|-----------------------:|----------------------:|-------------------:|---------------:|-----------------:|-----------------:|-----------------:|-------------------------:|-----------------------:|--------------------------:|----------------------------------:|