|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-similarity |
|
- sentence-transformers |
|
license: mit |
|
language: |
|
- multilingual |
|
- af |
|
- am |
|
- ar |
|
- as |
|
- az |
|
- be |
|
- bg |
|
- bn |
|
- br |
|
- bs |
|
- ca |
|
- cs |
|
- cy |
|
- da |
|
- de |
|
- el |
|
- en |
|
- eo |
|
- es |
|
- et |
|
- eu |
|
- fa |
|
- fi |
|
- fr |
|
- fy |
|
- ga |
|
- gd |
|
- gl |
|
- gu |
|
- ha |
|
- he |
|
- hi |
|
- hr |
|
- hu |
|
- hy |
|
- id |
|
- is |
|
- it |
|
- ja |
|
- jv |
|
- ka |
|
- kk |
|
- km |
|
- kn |
|
- ko |
|
- ku |
|
- ky |
|
- la |
|
- lo |
|
- lt |
|
- lv |
|
- mg |
|
- mk |
|
- ml |
|
- mn |
|
- mr |
|
- ms |
|
- my |
|
- ne |
|
- nl |
|
- no |
|
- om |
|
- or |
|
- pa |
|
- pl |
|
- ps |
|
- pt |
|
- ro |
|
- ru |
|
- sa |
|
- sd |
|
- si |
|
- sk |
|
- sl |
|
- so |
|
- sq |
|
- sr |
|
- su |
|
- sv |
|
- sw |
|
- ta |
|
- te |
|
- th |
|
- tl |
|
- tr |
|
- ug |
|
- uk |
|
- ur |
|
- uz |
|
- vi |
|
- xh |
|
- yi |
|
- zh |
|
--- |
|
|
|
A quantized version of [multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small). Quantization was performed per-layer under the same conditions as our ELSERv2 model, as described [here](https://www.elastic.co/search-labs/blog/articles/introducing-elser-v2-part-1#quantization). |
|
|
|
[Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf). |
|
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022 |
|
|
|
## Benchmarks |
|
|
|
We performed a number of small benchmarks to assess both the changes in quality as well as inference latency against the baseline original model. |
|
|
|
### Quality |
|
|
|
Measuring NDCG@10 using the dev split of the MIRACL datasets for select languages, we see mostly a marginal change in quality of the quantized model. |
|
|
|
| | de | yo| ru | ar | es | th | |
|
| --- | --- | ---| --- | --- | --- | --- | |
|
| multilingual-e5-small | 0.75862 | 0.56193 | 0.80309 | 0.82778 | 0.81672 | 0.85072 | |
|
| multilingual-e5-small-optimized | 0.75992 | 0.48934 | 0.79668 | 0.82017 | 0.8135 | 0.84316 | |
|
|
|
To test the English out-of-domain performance, we used the test split of various datasets in the BEIR evaluation. Measuring NDCG@10, we see a larger change in SCIFACT, but marginal in the other datasets evaluated. |
|
|
|
| | FIQA | SCIFACT | nfcorpus | |
|
| --- | --- | --- | --- | |
|
| multilingual-e5-small | 0.33126 | 0.677 | 0.31004 | |
|
| multilingual-e5-small-optimized | 0.31734 | 0.65484 | 0.30126 | |
|
|
|
### Performance |
|
|
|
Using a PyTorch model traced for Linux and Intel CPUs, we performed performance benchmarking with various lengths of input. Overall, we see on average a 50-20% performance improvement with the optimized model. |
|
|
|
| input length (characters) | multilingual-e5-small | multilingual-e5-small-optimized | speedup | |
|
| --- | --- | --- | --- | |
|
| 0 - 50 | 0.0181 | 0.00826 | 54.36% | |
|
| 50 - 100 | 0.0275 | 0.0164 | 40.36% | |
|
| 100 - 150 | 0.0366 | 0.0237 | 35.25% | |
|
| 150 - 200 | 0.0435 | 0.0301 | 30.80% | |
|
| 200 - 250 | 0.0514 | 0.0379 | 26.26% | |
|
| 250 - 300 | 0.0569 | 0.043 | 24.43% | |
|
| 300 - 350 | 0.0663 | 0.0513 | 22.62% | |
|
| 350 - 400 | 0.0737 | 0.0576 | 21.85% | |
|
|
|
### Disclaimer |
|
|
|
This e5 model, as defined, hosted, integrated and used in conjunction with our other Elastic Software is covered by our standard warranty. |
|
|
