joshdevins
commited on
Commit
•
26519d8
1
Parent(s):
d6ab683
Fix tables
Browse files
README.md
CHANGED
@@ -117,12 +117,14 @@ We performed a number of small benchmarks to assess both the changes in quality
|
|
117 |
Measuring NDCG@10 using the dev split of the MIRACL datasets for select languages, we see mostly a marginal change in quality of the quantized model.
|
118 |
|
119 |
| | de | yo| ru | ar | es | th |
|
|
|
120 |
| multilingual-e5-small | 0.75862 | 0.56193 | 0.80309 | 0.82778 | 0.81672 | 0.85072 |
|
121 |
| multilingual-e5-small-optimized | 0.75992 | 0.48934 | 0.79668 | 0.82017 | 0.8135 | 0.84316 |
|
122 |
|
123 |
To test the English out-of-domain performance, we used the test split of various datasets in the BEIR evaluation. Measuring NDCG@10, we see a larger changein SCIFACT, but marginal in the other datasets evaluated.
|
124 |
|
125 |
| | FIQA | SCIFACT | nfcorpus |
|
|
|
126 |
| multilingual-e5-small | 0.33126 | 0.677 | 0.31004 |
|
127 |
| multilingual-e5-small-optimized | 0.31734 | 0.65484 | 0.30126 |
|
128 |
|
@@ -131,6 +133,7 @@ To test the English out-of-domain performance, we used the test split of various
|
|
131 |
Using a PyTorch model traced for Linux and Intel CPUs, we performed performance benchmarking with various lengths of input. Overall, we see on average a 50-20% performance improvement with the optimized model.
|
132 |
|
133 |
| input length (characters) | multilingual-e5-small | multilingual-e5-small-optimized | speedup |
|
|
|
134 |
| 0 - 50 | 0.0181 | 0.00826 | 54.36% |
|
135 |
| 50 - 100 | 0.0275 | 0.0164 | 40.36% |
|
136 |
| 100 - 150 | 0.0366 | 0.0237 | 35.25% |
|
|
|
117 |
Measuring NDCG@10 using the dev split of the MIRACL datasets for select languages, we see mostly a marginal change in quality of the quantized model.
|
118 |
|
119 |
| | de | yo| ru | ar | es | th |
|
120 |
+
| --- | --- | ---| --- | --- | --- | --- |
|
121 |
| multilingual-e5-small | 0.75862 | 0.56193 | 0.80309 | 0.82778 | 0.81672 | 0.85072 |
|
122 |
| multilingual-e5-small-optimized | 0.75992 | 0.48934 | 0.79668 | 0.82017 | 0.8135 | 0.84316 |
|
123 |
|
124 |
To test the English out-of-domain performance, we used the test split of various datasets in the BEIR evaluation. Measuring NDCG@10, we see a larger changein SCIFACT, but marginal in the other datasets evaluated.
|
125 |
|
126 |
| | FIQA | SCIFACT | nfcorpus |
|
127 |
+
| --- | --- | --- | --- |
|
128 |
| multilingual-e5-small | 0.33126 | 0.677 | 0.31004 |
|
129 |
| multilingual-e5-small-optimized | 0.31734 | 0.65484 | 0.30126 |
|
130 |
|
|
|
133 |
Using a PyTorch model traced for Linux and Intel CPUs, we performed performance benchmarking with various lengths of input. Overall, we see on average a 50-20% performance improvement with the optimized model.
|
134 |
|
135 |
| input length (characters) | multilingual-e5-small | multilingual-e5-small-optimized | speedup |
|
136 |
+
| --- | --- | --- | --- |
|
137 |
| 0 - 50 | 0.0181 | 0.00826 | 54.36% |
|
138 |
| 50 - 100 | 0.0275 | 0.0164 | 40.36% |
|
139 |
| 100 - 150 | 0.0366 | 0.0237 | 35.25% |
|