Hugging Face's logo

philipp-zettl
/
MiniLM-similarity-small

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
Generated from Trainer
dataset_size:1267
loss:CoSENTLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
philipp-zettl commited on
Commit
3822642
•
1 Parent(s): 594a251

Add new SentenceTransformer model.

Browse files
Files changed (2) hide show
  1. README.md +98 -93
  2. model.safetensors +1 -1
README.md CHANGED
@@ -6,7 +6,7 @@ tags:
6
  - sentence-similarity
7
  - feature-extraction
8
  - generated_from_trainer
9
- - dataset_size:665
10
  - loss:CoSENTLoss
11
  base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
12
  datasets: []
@@ -22,31 +22,31 @@ metrics:
22
  - pearson_max
23
  - spearman_max
24
  widget:
25
- - source_sentence: Is there a free return policy?
26
  sentences:
27
- - general query
 
28
  - faq query
29
- - product query
30
- - source_sentence: Quiero reservar un vuelo a Madrid
31
  sentences:
32
- - faq query
33
- - general query
34
- - product query
35
- - source_sentence: Bestell mir einen Bluetooth-Lautsprecher
36
  sentences:
37
- - faq query
38
- - general query
39
- - general query
40
- - source_sentence: Kann ich meinen Account auf Premium upgraden?
41
  sentences:
42
- - faq query
43
  - product query
44
- - faq query
45
- - source_sentence: Was kostet der Versand nach Kanada?
46
  sentences:
47
  - product query
48
- - faq query
49
- - faq query
50
  pipeline_tag: sentence-similarity
51
  model-index:
52
  - name: SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
@@ -59,34 +59,34 @@ model-index:
59
  type: MiniLM-dev
60
  metrics:
61
  - type: pearson_cosine
62
- value: 0.7060858093148971
63
  name: Pearson Cosine
64
  - type: spearman_cosine
65
- value: 0.7122657953703283
66
  name: Spearman Cosine
67
  - type: pearson_manhattan
68
- value: 0.5850353380261794
69
  name: Pearson Manhattan
70
  - type: spearman_manhattan
71
- value: 0.6010204119883696
72
  name: Spearman Manhattan
73
  - type: pearson_euclidean
74
- value: 0.5997691394008732
75
  name: Pearson Euclidean
76
  - type: spearman_euclidean
77
- value: 0.6079117189235353
78
  name: Spearman Euclidean
79
  - type: pearson_dot
80
- value: 0.7251159526734934
81
  name: Pearson Dot
82
  - type: spearman_dot
83
- value: 0.732939716175825
84
  name: Spearman Dot
85
  - type: pearson_max
86
- value: 0.7251159526734934
87
  name: Pearson Max
88
  - type: spearman_max
89
- value: 0.732939716175825
90
  name: Spearman Max
91
  - task:
92
  type: semantic-similarity
@@ -96,34 +96,34 @@ model-index:
96
  type: MiniLM-test
97
  metrics:
98
  - type: pearson_cosine
99
- value: 0.8232712880664017
100
  name: Pearson Cosine
101
  - type: spearman_cosine
102
- value: 0.822196399839697
103
  name: Spearman Cosine
104
  - type: pearson_manhattan
105
- value: 0.7831863345453927
106
  name: Pearson Manhattan
107
  - type: spearman_manhattan
108
- value: 0.8000293400400974
109
  name: Spearman Manhattan
110
  - type: pearson_euclidean
111
- value: 0.792921493930252
112
  name: Pearson Euclidean
113
  - type: spearman_euclidean
114
- value: 0.80506730817637
115
  name: Spearman Euclidean
116
  - type: pearson_dot
117
- value: 0.8011854727667188
118
  name: Pearson Dot
119
  - type: spearman_dot
120
- value: 0.8151432444489153
121
  name: Spearman Dot
122
  - type: pearson_max
123
- value: 0.8232712880664017
124
  name: Pearson Max
125
  - type: spearman_max
126
- value: 0.822196399839697
127
  name: Spearman Max
128
  ---
129
 
@@ -176,8 +176,8 @@ from sentence_transformers import SentenceTransformer
176
  model = SentenceTransformer("philipp-zettl/MiniLM-similarity-small")
177
  # Run inference
178
  sentences = [
179
- 'Was kostet der Versand nach Kanada?',
180
- 'faq query',
181
  'product query',
182
  ]
183
  embeddings = model.encode(sentences)
@@ -222,18 +222,18 @@ You can finetune this model on your own dataset.
222
  * Dataset: `MiniLM-dev`
223
  * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
224
 
225
- | Metric | Value |
226
- |:--------------------|:-----------|
227
- | pearson_cosine | 0.7061 |
228
- | **spearman_cosine** | **0.7123** |
229
- | pearson_manhattan | 0.585 |
230
- | spearman_manhattan | 0.601 |
231
- | pearson_euclidean | 0.5998 |
232
- | spearman_euclidean | 0.6079 |
233
- | pearson_dot | 0.7251 |
234
- | spearman_dot | 0.7329 |
235
- | pearson_max | 0.7251 |
236
- | spearman_max | 0.7329 |
237
 
238
  #### Semantic Similarity
239
  * Dataset: `MiniLM-test`
@@ -241,16 +241,16 @@ You can finetune this model on your own dataset.
241
 
242
  | Metric | Value |
243
  |:--------------------|:-----------|
244
- | pearson_cosine | 0.8233 |
245
- | **spearman_cosine** | **0.8222** |
246
- | pearson_manhattan | 0.7832 |
247
- | spearman_manhattan | 0.8 |
248
- | pearson_euclidean | 0.7929 |
249
- | spearman_euclidean | 0.8051 |
250
- | pearson_dot | 0.8012 |
251
- | spearman_dot | 0.8151 |
252
- | pearson_max | 0.8233 |
253
- | spearman_max | 0.8222 |
254
 
255
  <!--
256
  ## Bias, Risks and Limitations
@@ -271,19 +271,19 @@ You can finetune this model on your own dataset.
271
  #### Unnamed Dataset
272
 
273
 
274
- * Size: 665 training samples
275
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
276
  * Approximate statistics based on the first 1000 samples:
277
  | | sentence1 | sentence2 | score |
278
  |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:--------------------------------------------------------------|
279
  | type | string | string | float |
280
- | details | <ul><li>min: 7 tokens</li><li>mean: 11.29 tokens</li><li>max: 19 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 5.31 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.5</li><li>max: 1.0</li></ul> |
281
  * Samples:
282
- | sentence1 | sentence2 | score |
283
- |:---------------------------------------------------------------------|:---------------------------|:-----------------|
284
- | <code>Send me deals on gaming accessories</code> | <code>product query</code> | <code>1.0</code> |
285
- | <code>Aidez-moi à synchroniser mes contacts sur mon téléphone</code> | <code>faq query</code> | <code>0.0</code> |
286
- | <code>Какие у вас есть предложения по ноутбукам?</code> | <code>faq query</code> | <code>0.0</code> |
287
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
288
  ```json
289
  {
@@ -297,19 +297,19 @@ You can finetune this model on your own dataset.
297
  #### Unnamed Dataset
298
 
299
 
300
- * Size: 84 evaluation samples
301
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
302
  * Approximate statistics based on the first 1000 samples:
303
  | | sentence1 | sentence2 | score |
304
  |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
305
  | type | string | string | float |
306
- | details | <ul><li>min: 7 tokens</li><li>mean: 11.32 tokens</li><li>max: 17 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 5.42 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.46</li><li>max: 1.0</li></ul> |
307
  * Samples:
308
- | sentence1 | sentence2 | score |
309
- |:-----------------------------------------------------------------------------|:---------------------------|:-----------------|
310
- | <code>كيف يمكنني تتبع شحنتي؟</code> | <code>support query</code> | <code>0.0</code> |
311
- | <code>Aidez-moi à configurer une nouvelle adresse e-mail</code> | <code>support query</code> | <code>1.0</code> |
312
- | <code>Envoyez-moi les dernières promotions sur les montres connectées</code> | <code>product query</code> | <code>1.0</code> |
313
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
314
  ```json
315
  {
@@ -447,23 +447,28 @@ You can finetune this model on your own dataset.
447
  ### Training Logs
448
  | Epoch | Step | Training Loss | loss | MiniLM-dev_spearman_cosine | MiniLM-test_spearman_cosine |
449
  |:------:|:----:|:-------------:|:------:|:--------------------------:|:---------------------------:|
450
- | 0.4762 | 10 | 1.3639 | 0.8946 | 0.0665 | - |
451
- | 0.9524 | 20 | 0.8488 | 0.7608 | 0.2318 | - |
452
- | 1.4286 | 30 | 0.6629 | 1.0463 | 0.3736 | - |
453
- | 1.9048 | 40 | 1.1413 | 1.1547 | 0.4159 | - |
454
- | 2.3810 | 50 | 1.8156 | 1.2059 | 0.4760 | - |
455
- | 2.8571 | 60 | 2.0179 | 0.8129 | 0.5794 | - |
456
- | 3.3333 | 70 | 0.3202 | 0.6236 | 0.6217 | - |
457
- | 3.8095 | 80 | 0.1437 | 0.6061 | 0.6404 | - |
458
- | 4.2857 | 90 | 1.1623 | 0.7312 | 0.6424 | - |
459
- | 4.7619 | 100 | 0.4376 | 0.5987 | 0.6621 | - |
460
- | 5.2381 | 110 | 0.5832 | 0.4848 | 0.6837 | - |
461
- | 5.7143 | 120 | 0.1749 | 0.3367 | 0.6896 | - |
462
- | 6.1905 | 130 | 0.0192 | 0.2607 | 0.6936 | - |
463
- | 6.6667 | 140 | 0.2047 | 0.2564 | 0.6995 | - |
464
- | 7.1429 | 150 | 0.404 | 0.2747 | 0.7103 | - |
465
- | 7.6190 | 160 | 0.0008 | 0.2764 | 0.7123 | - |
466
- | 8.0 | 168 | - | - | - | 0.8222 |
 
 
 
 
 
467
 
468
 
469
  ### Framework Versions
 
6
  - sentence-similarity
7
  - feature-extraction
8
  - generated_from_trainer
9
+ - dataset_size:844
10
  - loss:CoSENTLoss
11
  base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
12
  datasets: []
 
22
  - pearson_max
23
  - spearman_max
24
  widget:
25
+ - source_sentence: Hilf mir, das Software-Update durchzuführen
26
  sentences:
27
+ - order query
28
+ - support query
29
  - faq query
30
+ - source_sentence: 马上给我提供这个商品的跟踪信息
 
31
  sentences:
32
+ - payment query
33
+ - technical support query
34
+ - support query
35
+ - source_sentence: Downgrade my subscription plan
36
  sentences:
37
+ - support query
38
+ - product query
39
+ - product query
40
+ - source_sentence: Help resolve issues with my operating system
41
  sentences:
42
+ - technical support query
43
  - product query
44
+ - product query
45
+ - source_sentence: Ayúdame a solucionar problemas de red
46
  sentences:
47
  - product query
48
+ - support query
49
+ - product query
50
  pipeline_tag: sentence-similarity
51
  model-index:
52
  - name: SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
 
59
  type: MiniLM-dev
60
  metrics:
61
  - type: pearson_cosine
62
+ value: 0.7960441122484267
63
  name: Pearson Cosine
64
  - type: spearman_cosine
65
+ value: 0.8189711310679958
66
  name: Spearman Cosine
67
  - type: pearson_manhattan
68
+ value: 0.6824455970208276
69
  name: Pearson Manhattan
70
  - type: spearman_manhattan
71
+ value: 0.701004701178111
72
  name: Spearman Manhattan
73
  - type: pearson_euclidean
74
+ value: 0.6821384996384094
75
  name: Pearson Euclidean
76
  - type: spearman_euclidean
77
+ value: 0.7065633287645454
78
  name: Spearman Euclidean
79
  - type: pearson_dot
80
+ value: 0.7871337514786776
81
  name: Pearson Dot
82
  - type: spearman_dot
83
+ value: 0.7979718712970215
84
  name: Spearman Dot
85
  - type: pearson_max
86
+ value: 0.7960441122484267
87
  name: Pearson Max
88
  - type: spearman_max
89
+ value: 0.8189711310679958
90
  name: Spearman Max
91
  - task:
92
  type: semantic-similarity
 
96
  type: MiniLM-test
97
  metrics:
98
  - type: pearson_cosine
99
+ value: 0.7614418952584415
100
  name: Pearson Cosine
101
  - type: spearman_cosine
102
+ value: 0.7585961676423125
103
  name: Spearman Cosine
104
  - type: pearson_manhattan
105
+ value: 0.620319727073133
106
  name: Pearson Manhattan
107
  - type: spearman_manhattan
108
+ value: 0.6192118311486844
109
  name: Spearman Manhattan
110
  - type: pearson_euclidean
111
+ value: 0.6116132687052156
112
  name: Pearson Euclidean
113
  - type: spearman_euclidean
114
+ value: 0.6124276377795256
115
  name: Spearman Euclidean
116
  - type: pearson_dot
117
+ value: 0.7670292333817905
118
  name: Pearson Dot
119
  - type: spearman_dot
120
+ value: 0.7764817683428225
121
  name: Spearman Dot
122
  - type: pearson_max
123
+ value: 0.7670292333817905
124
  name: Pearson Max
125
  - type: spearman_max
126
+ value: 0.7764817683428225
127
  name: Spearman Max
128
  ---
129
 
 
176
  model = SentenceTransformer("philipp-zettl/MiniLM-similarity-small")
177
  # Run inference
178
  sentences = [
179
+ 'Ayúdame a solucionar problemas de red',
180
+ 'support query',
181
  'product query',
182
  ]
183
  embeddings = model.encode(sentences)
 
222
  * Dataset: `MiniLM-dev`
223
  * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
224
 
225
+ | Metric | Value |
226
+ |:--------------------|:----------|
227
+ | pearson_cosine | 0.796 |
228
+ | **spearman_cosine** | **0.819** |
229
+ | pearson_manhattan | 0.6824 |
230
+ | spearman_manhattan | 0.701 |
231
+ | pearson_euclidean | 0.6821 |
232
+ | spearman_euclidean | 0.7066 |
233
+ | pearson_dot | 0.7871 |
234
+ | spearman_dot | 0.798 |
235
+ | pearson_max | 0.796 |
236
+ | spearman_max | 0.819 |
237
 
238
  #### Semantic Similarity
239
  * Dataset: `MiniLM-test`
 
241
 
242
  | Metric | Value |
243
  |:--------------------|:-----------|
244
+ | pearson_cosine | 0.7614 |
245
+ | **spearman_cosine** | **0.7586** |
246
+ | pearson_manhattan | 0.6203 |
247
+ | spearman_manhattan | 0.6192 |
248
+ | pearson_euclidean | 0.6116 |
249
+ | spearman_euclidean | 0.6124 |
250
+ | pearson_dot | 0.767 |
251
+ | spearman_dot | 0.7765 |
252
+ | pearson_max | 0.767 |
253
+ | spearman_max | 0.7765 |
254
 
255
  <!--
256
  ## Bias, Risks and Limitations
 
271
  #### Unnamed Dataset
272
 
273
 
274
+ * Size: 844 training samples
275
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
276
  * Approximate statistics based on the first 1000 samples:
277
  | | sentence1 | sentence2 | score |
278
  |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:--------------------------------------------------------------|
279
  | type | string | string | float |
280
+ | details | <ul><li>min: 6 tokens</li><li>mean: 10.83 tokens</li><li>max: 19 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.34 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.5</li><li>max: 1.0</li></ul> |
281
  * Samples:
282
+ | sentence1 | sentence2 | score |
283
+ |:-----------------------------------------------------------------|:---------------------------|:-----------------|
284
+ | <code>Покажите мне доступные гостиницы в Москве</code> | <code>product query</code> | <code>1.0</code> |
285
+ | <code>أرني العروض المتاحة على الهواتف الذكية</code> | <code>product query</code> | <code>1.0</code> |
286
+ | <code>Tengo problemas con el micrófono, ¿puedes ayudarme?</code> | <code>product query</code> | <code>0.0</code> |
287
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
288
  ```json
289
  {
 
297
  #### Unnamed Dataset
298
 
299
 
300
+ * Size: 106 evaluation samples
301
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
302
  * Approximate statistics based on the first 1000 samples:
303
  | | sentence1 | sentence2 | score |
304
  |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
305
  | type | string | string | float |
306
+ | details | <ul><li>min: 7 tokens</li><li>mean: 10.63 tokens</li><li>max: 17 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.32 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.47</li><li>max: 1.0</li></ul> |
307
  * Samples:
308
+ | sentence1 | sentence2 | score |
309
+ |:---------------------------------------------------------|:---------------------------|:-----------------|
310
+ | <code>Help me with device driver installation</code> | <code>product query</code> | <code>0.0</code> |
311
+ | <code>Check the status of my account verification</code> | <code>product query</code> | <code>0.0</code> |
312
+ | <code>我怎样重置我的密码？</code> | <code>product query</code> | <code>0.0</code> |
313
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
314
  ```json
315
  {
 
447
  ### Training Logs
448
  | Epoch | Step | Training Loss | loss | MiniLM-dev_spearman_cosine | MiniLM-test_spearman_cosine |
449
  |:------:|:----:|:-------------:|:------:|:--------------------------:|:---------------------------:|
450
+ | 0.3704 | 10 | 5.613 | 1.4994 | 0.2761 | - |
451
+ | 0.7407 | 20 | 4.8872 | 1.3690 | 0.3483 | - |
452
+ | 1.1111 | 30 | 3.2993 | 1.0579 | 0.4657 | - |
453
+ | 1.4815 | 40 | 2.1968 | 0.6858 | 0.5935 | - |
454
+ | 1.8519 | 50 | 0.7306 | 0.5191 | 0.6528 | - |
455
+ | 2.2222 | 60 | 0.9746 | 0.3735 | 0.6998 | - |
456
+ | 2.5926 | 70 | 0.3889 | 0.3532 | 0.7393 | - |
457
+ | 2.9630 | 80 | 0.1857 | 0.3598 | 0.7554 | - |
458
+ | 3.3333 | 90 | 0.2923 | 0.2795 | 0.7714 | - |
459
+ | 3.7037 | 100 | 0.6776 | 0.2881 | 0.7825 | - |
460
+ | 4.0741 | 110 | 0.2404 | 0.2679 | 0.7887 | - |
461
+ | 4.4444 | 120 | 0.0168 | 0.2583 | 0.7918 | - |
462
+ | 4.8148 | 130 | 0.0179 | 0.2273 | 0.7980 | - |
463
+ | 5.1852 | 140 | 0.0006 | 0.2196 | 0.8023 | - |
464
+ | 5.5556 | 150 | 0.0276 | 0.2068 | 0.8066 | - |
465
+ | 5.9259 | 160 | 0.061 | 0.2063 | 0.8103 | - |
466
+ | 6.2963 | 170 | 0.0265 | 0.2259 | 0.8103 | - |
467
+ | 6.6667 | 180 | 0.0105 | 0.2236 | 0.8165 | - |
468
+ | 7.0370 | 190 | 0.0008 | 0.2208 | 0.8177 | - |
469
+ | 7.4074 | 200 | 0.361 | 0.2340 | 0.8171 | - |
470
+ | 7.7778 | 210 | 0.0 | 0.2345 | 0.8190 | - |
471
+ | 8.0 | 216 | - | - | - | 0.7586 |
472
 
473
 
474
  ### Framework Versions
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f93e7a04af4cede03ff8facfb35e603c3269ab0d128fa7c1216b44d2ed3b40f9
3
  size 470637416
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f39ac4242378e0521f1f13befca327f07a17029f5b7262ca4a7f5dcd050d435
3
  size 470637416