joshdevins commited on
Commit
26519d8
1 Parent(s): d6ab683

Fix tables

Browse files
Files changed (1) hide show
  1. README.md +3 -0
README.md CHANGED
@@ -117,12 +117,14 @@ We performed a number of small benchmarks to assess both the changes in quality
117
  Measuring NDCG@10 using the dev split of the MIRACL datasets for select languages, we see mostly a marginal change in quality of the quantized model.
118
 
119
  | | de | yo| ru | ar | es | th |
 
120
  | multilingual-e5-small | 0.75862 | 0.56193 | 0.80309 | 0.82778 | 0.81672 | 0.85072 |
121
  | multilingual-e5-small-optimized | 0.75992 | 0.48934 | 0.79668 | 0.82017 | 0.8135 | 0.84316 |
122
 
123
  To test the English out-of-domain performance, we used the test split of various datasets in the BEIR evaluation. Measuring NDCG@10, we see a larger changein SCIFACT, but marginal in the other datasets evaluated.
124
 
125
  | | FIQA | SCIFACT | nfcorpus |
 
126
  | multilingual-e5-small | 0.33126 | 0.677 | 0.31004 |
127
  | multilingual-e5-small-optimized | 0.31734 | 0.65484 | 0.30126 |
128
 
@@ -131,6 +133,7 @@ To test the English out-of-domain performance, we used the test split of various
131
  Using a PyTorch model traced for Linux and Intel CPUs, we performed performance benchmarking with various lengths of input. Overall, we see on average a 50-20% performance improvement with the optimized model.
132
 
133
  | input length (characters) | multilingual-e5-small | multilingual-e5-small-optimized | speedup |
 
134
  | 0 - 50 | 0.0181 | 0.00826 | 54.36% |
135
  | 50 - 100 | 0.0275 | 0.0164 | 40.36% |
136
  | 100 - 150 | 0.0366 | 0.0237 | 35.25% |
 
117
  Measuring NDCG@10 using the dev split of the MIRACL datasets for select languages, we see mostly a marginal change in quality of the quantized model.
118
 
119
  | | de | yo| ru | ar | es | th |
120
+ | --- | --- | ---| --- | --- | --- | --- |
121
  | multilingual-e5-small | 0.75862 | 0.56193 | 0.80309 | 0.82778 | 0.81672 | 0.85072 |
122
  | multilingual-e5-small-optimized | 0.75992 | 0.48934 | 0.79668 | 0.82017 | 0.8135 | 0.84316 |
123
 
124
  To test the English out-of-domain performance, we used the test split of various datasets in the BEIR evaluation. Measuring NDCG@10, we see a larger changein SCIFACT, but marginal in the other datasets evaluated.
125
 
126
  | | FIQA | SCIFACT | nfcorpus |
127
+ | --- | --- | --- | --- |
128
  | multilingual-e5-small | 0.33126 | 0.677 | 0.31004 |
129
  | multilingual-e5-small-optimized | 0.31734 | 0.65484 | 0.30126 |
130
 
 
133
  Using a PyTorch model traced for Linux and Intel CPUs, we performed performance benchmarking with various lengths of input. Overall, we see on average a 50-20% performance improvement with the optimized model.
134
 
135
  | input length (characters) | multilingual-e5-small | multilingual-e5-small-optimized | speedup |
136
+ | --- | --- | --- | --- |
137
  | 0 - 50 | 0.0181 | 0.00826 | 54.36% |
138
  | 50 - 100 | 0.0275 | 0.0164 | 40.36% |
139
  | 100 - 150 | 0.0366 | 0.0237 | 35.25% |