Yinka / README.md
Classical's picture
Upload README.md
b1226e2 verified
|
raw
history blame
26.6 kB
metadata
tags:
  - mteb
model-index:
  - name: checkpoint-1431
    results:
      - task:
          type: STS
        dataset:
          type: C-MTEB/AFQMC
          name: MTEB AFQMC
          config: default
          split: validation
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 56.306314279047875
          - type: cos_sim_spearman
            value: 61.020227685004016
          - type: euclidean_pearson
            value: 58.61821670933433
          - type: euclidean_spearman
            value: 60.131457106640674
          - type: manhattan_pearson
            value: 58.6189460369694
          - type: manhattan_spearman
            value: 60.126350618526224
      - task:
          type: STS
        dataset:
          type: C-MTEB/ATEC
          name: MTEB ATEC
          config: default
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 55.8612958476143
          - type: cos_sim_spearman
            value: 59.01977664864512
          - type: euclidean_pearson
            value: 62.028094897243655
          - type: euclidean_spearman
            value: 58.6046814257705
          - type: manhattan_pearson
            value: 62.02580042431887
          - type: manhattan_spearman
            value: 58.60626890004892
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (zh)
          config: zh
          split: test
          revision: 1399c76144fd37290681b995c656ef9b2e06e26d
        metrics:
          - type: accuracy
            value: 49.496
          - type: f1
            value: 46.673963383873065
      - task:
          type: STS
        dataset:
          type: C-MTEB/BQ
          name: MTEB BQ
          config: default
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 70.73971622592535
          - type: cos_sim_spearman
            value: 72.76102992060764
          - type: euclidean_pearson
            value: 71.04525865868672
          - type: euclidean_spearman
            value: 72.4032852155075
          - type: manhattan_pearson
            value: 71.03693009336658
          - type: manhattan_spearman
            value: 72.39635701224252
      - task:
          type: Clustering
        dataset:
          type: C-MTEB/CLSClusteringP2P
          name: MTEB CLSClusteringP2P
          config: default
          split: test
          revision: None
        metrics:
          - type: v_measure
            value: 56.34751074520767
      - task:
          type: Clustering
        dataset:
          type: C-MTEB/CLSClusteringS2S
          name: MTEB CLSClusteringS2S
          config: default
          split: test
          revision: None
        metrics:
          - type: v_measure
            value: 48.4856662121073
      - task:
          type: Reranking
        dataset:
          type: C-MTEB/CMedQAv1-reranking
          name: MTEB CMedQAv1
          config: default
          split: test
          revision: None
        metrics:
          - type: map
            value: 89.26384109024997
          - type: mrr
            value: 91.27261904761905
      - task:
          type: Reranking
        dataset:
          type: C-MTEB/CMedQAv2-reranking
          name: MTEB CMedQAv2
          config: default
          split: test
          revision: None
        metrics:
          - type: map
            value: 90.0464058154547
          - type: mrr
            value: 92.06480158730159
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/CmedqaRetrieval
          name: MTEB CmedqaRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 27.236
          - type: map_at_10
            value: 40.778
          - type: map_at_100
            value: 42.692
          - type: map_at_1000
            value: 42.787
          - type: map_at_3
            value: 36.362
          - type: map_at_5
            value: 38.839
          - type: mrr_at_1
            value: 41.335
          - type: mrr_at_10
            value: 49.867
          - type: mrr_at_100
            value: 50.812999999999995
          - type: mrr_at_1000
            value: 50.848000000000006
          - type: mrr_at_3
            value: 47.354
          - type: mrr_at_5
            value: 48.718
          - type: ndcg_at_1
            value: 41.335
          - type: ndcg_at_10
            value: 47.642
          - type: ndcg_at_100
            value: 54.855
          - type: ndcg_at_1000
            value: 56.449000000000005
          - type: ndcg_at_3
            value: 42.203
          - type: ndcg_at_5
            value: 44.416
          - type: precision_at_1
            value: 41.335
          - type: precision_at_10
            value: 10.568
          - type: precision_at_100
            value: 1.6400000000000001
          - type: precision_at_1000
            value: 0.184
          - type: precision_at_3
            value: 23.998
          - type: precision_at_5
            value: 17.389
          - type: recall_at_1
            value: 27.236
          - type: recall_at_10
            value: 58.80800000000001
          - type: recall_at_100
            value: 88.411
          - type: recall_at_1000
            value: 99.032
          - type: recall_at_3
            value: 42.253
          - type: recall_at_5
            value: 49.118
      - task:
          type: PairClassification
        dataset:
          type: C-MTEB/CMNLI
          name: MTEB Cmnli
          config: default
          split: validation
          revision: None
        metrics:
          - type: cos_sim_accuracy
            value: 86.03728202044498
          - type: cos_sim_ap
            value: 92.49469583272597
          - type: cos_sim_f1
            value: 86.74095974528088
          - type: cos_sim_precision
            value: 84.43657294664601
          - type: cos_sim_recall
            value: 89.17465513210195
          - type: dot_accuracy
            value: 72.21888153938664
          - type: dot_ap
            value: 80.59377163340332
          - type: dot_f1
            value: 74.96686040583258
          - type: dot_precision
            value: 66.4737793851718
          - type: dot_recall
            value: 85.94809445873275
          - type: euclidean_accuracy
            value: 85.47203848466627
          - type: euclidean_ap
            value: 91.89152584749868
          - type: euclidean_f1
            value: 86.38105975197294
          - type: euclidean_precision
            value: 83.40953625081646
          - type: euclidean_recall
            value: 89.5721299976619
          - type: manhattan_accuracy
            value: 85.3758268190018
          - type: manhattan_ap
            value: 91.88989707722311
          - type: manhattan_f1
            value: 86.39767519839052
          - type: manhattan_precision
            value: 82.76231263383298
          - type: manhattan_recall
            value: 90.36707972878185
          - type: max_accuracy
            value: 86.03728202044498
          - type: max_ap
            value: 92.49469583272597
          - type: max_f1
            value: 86.74095974528088
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/CovidRetrieval
          name: MTEB CovidRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 74.34100000000001
          - type: map_at_10
            value: 82.49499999999999
          - type: map_at_100
            value: 82.64200000000001
          - type: map_at_1000
            value: 82.643
          - type: map_at_3
            value: 81.142
          - type: map_at_5
            value: 81.95400000000001
          - type: mrr_at_1
            value: 74.71
          - type: mrr_at_10
            value: 82.553
          - type: mrr_at_100
            value: 82.699
          - type: mrr_at_1000
            value: 82.70100000000001
          - type: mrr_at_3
            value: 81.279
          - type: mrr_at_5
            value: 82.069
          - type: ndcg_at_1
            value: 74.605
          - type: ndcg_at_10
            value: 85.946
          - type: ndcg_at_100
            value: 86.607
          - type: ndcg_at_1000
            value: 86.669
          - type: ndcg_at_3
            value: 83.263
          - type: ndcg_at_5
            value: 84.71600000000001
          - type: precision_at_1
            value: 74.605
          - type: precision_at_10
            value: 9.758
          - type: precision_at_100
            value: 1.005
          - type: precision_at_1000
            value: 0.101
          - type: precision_at_3
            value: 29.996000000000002
          - type: precision_at_5
            value: 18.736
          - type: recall_at_1
            value: 74.34100000000001
          - type: recall_at_10
            value: 96.523
          - type: recall_at_100
            value: 99.473
          - type: recall_at_1000
            value: 100
          - type: recall_at_3
            value: 89.278
          - type: recall_at_5
            value: 92.83500000000001
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/DuRetrieval
          name: MTEB DuRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 26.950000000000003
          - type: map_at_10
            value: 82.408
          - type: map_at_100
            value: 85.057
          - type: map_at_1000
            value: 85.09100000000001
          - type: map_at_3
            value: 57.635999999999996
          - type: map_at_5
            value: 72.48
          - type: mrr_at_1
            value: 92.15
          - type: mrr_at_10
            value: 94.554
          - type: mrr_at_100
            value: 94.608
          - type: mrr_at_1000
            value: 94.61
          - type: mrr_at_3
            value: 94.292
          - type: mrr_at_5
            value: 94.459
          - type: ndcg_at_1
            value: 92.15
          - type: ndcg_at_10
            value: 89.108
          - type: ndcg_at_100
            value: 91.525
          - type: ndcg_at_1000
            value: 91.82900000000001
          - type: ndcg_at_3
            value: 88.44
          - type: ndcg_at_5
            value: 87.271
          - type: precision_at_1
            value: 92.15
          - type: precision_at_10
            value: 42.29
          - type: precision_at_100
            value: 4.812
          - type: precision_at_1000
            value: 0.48900000000000005
          - type: precision_at_3
            value: 79.14999999999999
          - type: precision_at_5
            value: 66.64
          - type: recall_at_1
            value: 26.950000000000003
          - type: recall_at_10
            value: 89.832
          - type: recall_at_100
            value: 97.921
          - type: recall_at_1000
            value: 99.471
          - type: recall_at_3
            value: 59.562000000000005
          - type: recall_at_5
            value: 76.533
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/EcomRetrieval
          name: MTEB EcomRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 53.5
          - type: map_at_10
            value: 63.105999999999995
          - type: map_at_100
            value: 63.63100000000001
          - type: map_at_1000
            value: 63.641999999999996
          - type: map_at_3
            value: 60.617
          - type: map_at_5
            value: 62.132
          - type: mrr_at_1
            value: 53.5
          - type: mrr_at_10
            value: 63.105999999999995
          - type: mrr_at_100
            value: 63.63100000000001
          - type: mrr_at_1000
            value: 63.641999999999996
          - type: mrr_at_3
            value: 60.617
          - type: mrr_at_5
            value: 62.132
          - type: ndcg_at_1
            value: 53.5
          - type: ndcg_at_10
            value: 67.92200000000001
          - type: ndcg_at_100
            value: 70.486
          - type: ndcg_at_1000
            value: 70.777
          - type: ndcg_at_3
            value: 62.853
          - type: ndcg_at_5
            value: 65.59899999999999
          - type: precision_at_1
            value: 53.5
          - type: precision_at_10
            value: 8.309999999999999
          - type: precision_at_100
            value: 0.951
          - type: precision_at_1000
            value: 0.097
          - type: precision_at_3
            value: 23.1
          - type: precision_at_5
            value: 15.2
          - type: recall_at_1
            value: 53.5
          - type: recall_at_10
            value: 83.1
          - type: recall_at_100
            value: 95.1
          - type: recall_at_1000
            value: 97.39999999999999
          - type: recall_at_3
            value: 69.3
          - type: recall_at_5
            value: 76
      - task:
          type: Classification
        dataset:
          type: C-MTEB/IFlyTek-classification
          name: MTEB IFlyTek
          config: default
          split: validation
          revision: None
        metrics:
          - type: accuracy
            value: 51.773759138130046
          - type: f1
            value: 40.38600802756481
      - task:
          type: Classification
        dataset:
          type: C-MTEB/JDReview-classification
          name: MTEB JDReview
          config: default
          split: test
          revision: None
        metrics:
          - type: accuracy
            value: 88.48030018761726
          - type: ap
            value: 59.2732541555627
          - type: f1
            value: 83.58836007358619
      - task:
          type: STS
        dataset:
          type: C-MTEB/LCQMC
          name: MTEB LCQMC
          config: default
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 73.67511194245922
          - type: cos_sim_spearman
            value: 79.43347759067298
          - type: euclidean_pearson
            value: 79.04491504318766
          - type: euclidean_spearman
            value: 79.14478545356785
          - type: manhattan_pearson
            value: 79.03365022867428
          - type: manhattan_spearman
            value: 79.13172717619908
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/MMarcoRetrieval
          name: MTEB MMarcoRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 67.184
          - type: map_at_10
            value: 76.24600000000001
          - type: map_at_100
            value: 76.563
          - type: map_at_1000
            value: 76.575
          - type: map_at_3
            value: 74.522
          - type: map_at_5
            value: 75.598
          - type: mrr_at_1
            value: 69.47
          - type: mrr_at_10
            value: 76.8
          - type: mrr_at_100
            value: 77.082
          - type: mrr_at_1000
            value: 77.093
          - type: mrr_at_3
            value: 75.29400000000001
          - type: mrr_at_5
            value: 76.24
          - type: ndcg_at_1
            value: 69.47
          - type: ndcg_at_10
            value: 79.81099999999999
          - type: ndcg_at_100
            value: 81.187
          - type: ndcg_at_1000
            value: 81.492
          - type: ndcg_at_3
            value: 76.536
          - type: ndcg_at_5
            value: 78.367
          - type: precision_at_1
            value: 69.47
          - type: precision_at_10
            value: 9.599
          - type: precision_at_100
            value: 1.026
          - type: precision_at_1000
            value: 0.105
          - type: precision_at_3
            value: 28.777
          - type: precision_at_5
            value: 18.232
          - type: recall_at_1
            value: 67.184
          - type: recall_at_10
            value: 90.211
          - type: recall_at_100
            value: 96.322
          - type: recall_at_1000
            value: 98.699
          - type: recall_at_3
            value: 81.556
          - type: recall_at_5
            value: 85.931
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (zh-CN)
          config: zh-CN
          split: test
          revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
        metrics:
          - type: accuracy
            value: 76.96032279757901
          - type: f1
            value: 73.48052314033545
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (zh-CN)
          config: zh-CN
          split: test
          revision: 7d571f92784cd94a019292a1f45445077d0ef634
        metrics:
          - type: accuracy
            value: 84.64357767316744
          - type: f1
            value: 83.58250539497922
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/MedicalRetrieval
          name: MTEB MedicalRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 56.00000000000001
          - type: map_at_10
            value: 62.066
          - type: map_at_100
            value: 62.553000000000004
          - type: map_at_1000
            value: 62.598
          - type: map_at_3
            value: 60.4
          - type: map_at_5
            value: 61.370000000000005
          - type: mrr_at_1
            value: 56.2
          - type: mrr_at_10
            value: 62.166
          - type: mrr_at_100
            value: 62.653000000000006
          - type: mrr_at_1000
            value: 62.699000000000005
          - type: mrr_at_3
            value: 60.5
          - type: mrr_at_5
            value: 61.47
          - type: ndcg_at_1
            value: 56.00000000000001
          - type: ndcg_at_10
            value: 65.199
          - type: ndcg_at_100
            value: 67.79899999999999
          - type: ndcg_at_1000
            value: 69.056
          - type: ndcg_at_3
            value: 61.814
          - type: ndcg_at_5
            value: 63.553000000000004
          - type: precision_at_1
            value: 56.00000000000001
          - type: precision_at_10
            value: 7.51
          - type: precision_at_100
            value: 0.878
          - type: precision_at_1000
            value: 0.098
          - type: precision_at_3
            value: 21.967
          - type: precision_at_5
            value: 14.02
          - type: recall_at_1
            value: 56.00000000000001
          - type: recall_at_10
            value: 75.1
          - type: recall_at_100
            value: 87.8
          - type: recall_at_1000
            value: 97.7
          - type: recall_at_3
            value: 65.9
          - type: recall_at_5
            value: 70.1
      - task:
          type: Reranking
        dataset:
          type: C-MTEB/Mmarco-reranking
          name: MTEB MMarcoReranking
          config: default
          split: dev
          revision: None
        metrics:
          - type: map
            value: 32.74158258279793
          - type: mrr
            value: 31.56071428571428
      - task:
          type: Classification
        dataset:
          type: C-MTEB/MultilingualSentiment-classification
          name: MTEB MultilingualSentiment
          config: default
          split: validation
          revision: None
        metrics:
          - type: accuracy
            value: 78.96666666666667
          - type: f1
            value: 78.82528563818045
      - task:
          type: PairClassification
        dataset:
          type: C-MTEB/OCNLI
          name: MTEB Ocnli
          config: default
          split: validation
          revision: None
        metrics:
          - type: cos_sim_accuracy
            value: 83.54087709799674
          - type: cos_sim_ap
            value: 87.26170197077586
          - type: cos_sim_f1
            value: 84.7609561752988
          - type: cos_sim_precision
            value: 80.20735155513667
          - type: cos_sim_recall
            value: 89.86272439281943
          - type: dot_accuracy
            value: 72.22523010286952
          - type: dot_ap
            value: 79.51975358187732
          - type: dot_f1
            value: 76.32183908045977
          - type: dot_precision
            value: 67.58957654723126
          - type: dot_recall
            value: 87.64519535374869
          - type: euclidean_accuracy
            value: 82.0249052517596
          - type: euclidean_ap
            value: 85.32829948726406
          - type: euclidean_f1
            value: 83.24924318869829
          - type: euclidean_precision
            value: 79.71014492753623
          - type: euclidean_recall
            value: 87.11721224920802
          - type: manhattan_accuracy
            value: 82.13318895506227
          - type: manhattan_ap
            value: 85.28856869288006
          - type: manhattan_f1
            value: 83.34946757018393
          - type: manhattan_precision
            value: 76.94369973190348
          - type: manhattan_recall
            value: 90.91869060190075
          - type: max_accuracy
            value: 83.54087709799674
          - type: max_ap
            value: 87.26170197077586
          - type: max_f1
            value: 84.7609561752988
      - task:
          type: Classification
        dataset:
          type: C-MTEB/OnlineShopping-classification
          name: MTEB OnlineShopping
          config: default
          split: test
          revision: None
        metrics:
          - type: accuracy
            value: 94.56
          - type: ap
            value: 92.80848436710805
          - type: f1
            value: 94.54951966576111
      - task:
          type: STS
        dataset:
          type: C-MTEB/PAWSX
          name: MTEB PAWSX
          config: default
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 39.0866558287863
          - type: cos_sim_spearman
            value: 45.9211126233312
          - type: euclidean_pearson
            value: 44.86568743222145
          - type: euclidean_spearman
            value: 45.63882757207507
          - type: manhattan_pearson
            value: 44.89480036909126
          - type: manhattan_spearman
            value: 45.65929449046206
      - task:
          type: STS
        dataset:
          type: C-MTEB/QBQTC
          name: MTEB QBQTC
          config: default
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 43.04701793979569
          - type: cos_sim_spearman
            value: 44.87491033760315
          - type: euclidean_pearson
            value: 36.2004061032567
          - type: euclidean_spearman
            value: 41.44823909683865
          - type: manhattan_pearson
            value: 36.136113427955095
          - type: manhattan_spearman
            value: 41.39225495993949
      - task:
          type: STS
        dataset:
          type: mteb/sts22-crosslingual-sts
          name: MTEB STS22 (zh)
          config: zh
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 61.65611315777857
          - type: cos_sim_spearman
            value: 64.4067673105648
          - type: euclidean_pearson
            value: 61.814977248797184
          - type: euclidean_spearman
            value: 63.99473350700169
          - type: manhattan_pearson
            value: 61.684304629588624
          - type: manhattan_spearman
            value: 63.97831213239316
      - task:
          type: STS
        dataset:
          type: C-MTEB/STSB
          name: MTEB STSB
          config: default
          split: test
          revision: None
        metrics:
          - type: cos_sim_pearson
            value: 76.57324933064379
          - type: cos_sim_spearman
            value: 79.23602286949782
          - type: euclidean_pearson
            value: 80.28226284310948
          - type: euclidean_spearman
            value: 80.32210477608423
          - type: manhattan_pearson
            value: 80.27262188617811
          - type: manhattan_spearman
            value: 80.31619185039723
      - task:
          type: Reranking
        dataset:
          type: C-MTEB/T2Reranking
          name: MTEB T2Reranking
          config: default
          split: dev
          revision: None
        metrics:
          - type: map
            value: 67.05266891356277
          - type: mrr
            value: 77.1906333623497
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/T2Retrieval
          name: MTEB T2Retrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 28.212
          - type: map_at_10
            value: 78.932
          - type: map_at_100
            value: 82.51899999999999
          - type: map_at_1000
            value: 82.575
          - type: map_at_3
            value: 55.614
          - type: map_at_5
            value: 68.304
          - type: mrr_at_1
            value: 91.211
          - type: mrr_at_10
            value: 93.589
          - type: mrr_at_100
            value: 93.659
          - type: mrr_at_1000
            value: 93.662
          - type: mrr_at_3
            value: 93.218
          - type: mrr_at_5
            value: 93.453
          - type: ndcg_at_1
            value: 91.211
          - type: ndcg_at_10
            value: 86.24000000000001
          - type: ndcg_at_100
            value: 89.614
          - type: ndcg_at_1000
            value: 90.14
          - type: ndcg_at_3
            value: 87.589
          - type: ndcg_at_5
            value: 86.265
          - type: precision_at_1
            value: 91.211
          - type: precision_at_10
            value: 42.626
          - type: precision_at_100
            value: 5.043
          - type: precision_at_1000
            value: 0.517
          - type: precision_at_3
            value: 76.42
          - type: precision_at_5
            value: 64.045
          - type: recall_at_1
            value: 28.212
          - type: recall_at_10
            value: 85.223
          - type: recall_at_100
            value: 96.229
          - type: recall_at_1000
            value: 98.849
          - type: recall_at_3
            value: 57.30800000000001
          - type: recall_at_5
            value: 71.661
      - task:
          type: Classification
        dataset:
          type: C-MTEB/TNews-classification
          name: MTEB TNews
          config: default
          split: validation
          revision: None
        metrics:
          - type: accuracy
            value: 54.385000000000005
          - type: f1
            value: 52.38762400903556
      - task:
          type: Clustering
        dataset:
          type: C-MTEB/ThuNewsClusteringP2P
          name: MTEB ThuNewsClusteringP2P
          config: default
          split: test
          revision: None
        metrics:
          - type: v_measure
            value: 74.55283855942916
      - task:
          type: Clustering
        dataset:
          type: C-MTEB/ThuNewsClusteringS2S
          name: MTEB ThuNewsClusteringS2S
          config: default
          split: test
          revision: None
        metrics:
          - type: v_measure
            value: 68.55115316700493
      - task:
          type: Retrieval
        dataset:
          type: C-MTEB/VideoRetrieval
          name: MTEB VideoRetrieval
          config: default
          split: dev
          revision: None
        metrics:
          - type: map_at_1
            value: 58.8
          - type: map_at_10
            value: 69.035
          - type: map_at_100
            value: 69.52000000000001
          - type: map_at_1000
            value: 69.529
          - type: map_at_3
            value: 67.417
          - type: map_at_5
            value: 68.407
          - type: mrr_at_1
            value: 58.8
          - type: mrr_at_10
            value: 69.035
          - type: mrr_at_100
            value: 69.52000000000001
          - type: mrr_at_1000
            value: 69.529
          - type: mrr_at_3
            value: 67.417
          - type: mrr_at_5
            value: 68.407
          - type: ndcg_at_1
            value: 58.8
          - type: ndcg_at_10
            value: 73.395
          - type: ndcg_at_100
            value: 75.62
          - type: ndcg_at_1000
            value: 75.90299999999999
          - type: ndcg_at_3
            value: 70.11800000000001
          - type: ndcg_at_5
            value: 71.87400000000001
          - type: precision_at_1
            value: 58.8
          - type: precision_at_10
            value: 8.68
          - type: precision_at_100
            value: 0.9690000000000001
          - type: precision_at_1000
            value: 0.099
          - type: precision_at_3
            value: 25.967000000000002
          - type: precision_at_5
            value: 16.42
          - type: recall_at_1
            value: 58.8
          - type: recall_at_10
            value: 86.8
          - type: recall_at_100
            value: 96.89999999999999
          - type: recall_at_1000
            value: 99.2
          - type: recall_at_3
            value: 77.9
          - type: recall_at_5
            value: 82.1
      - task:
          type: Classification
        dataset:
          type: C-MTEB/waimai-classification
          name: MTEB Waimai
          config: default
          split: test
          revision: None
        metrics:
          - type: accuracy
            value: 89.42
          - type: ap
            value: 75.35978503182068
          - type: f1
            value: 88.01006394348263

Yinka

Yinka embedding 模型是在开原模型stella-v3.5-mrl上续训的,采用了piccolo2提到的多任务混合损失(multi-task hybrid loss training)。同样本模型也支持了可变的向量维度。

使用方法

该模型的使用方法同stella-v3.5-mrl一样, 无需任何前缀。

from sentence_transformers import SentenceTransformer
from sklearn.preprocessing import normalize

model = SentenceTransformer("Classical/Yinka")
# 注意先不要normalize! 选取前n维后再normalize
vectors = model.encode(["text1", "text2"], normalize_embeddings=False)
print(vectors.shape)  # shape is [2,1792]
n_dims = 768
cut_vecs = normalize(vectors[:, :n_dims])

结果

Model Name Model Size (GB) Dimension Sequence Length Classification (9) Clustering (4) Pair Classification (2) Reranking (4) Retrieval (8) STS (8) Average (35)
Yinka 1.21 1792 512 74.30 61.99 89.87 69.77 74.40 63.30 70.79
stella-v3.5-mrl 1.21 1792 512 71.56 54.39 88.09 68.45 73.51 62.48 68.56
Yinka 1.21 1792 512 74.30 61.99 89.87 69.77 74.40 63.30 70.79
piccolo-large-zh-v2 1.21 1792 512 74.59 62.17 90.24 70 74.36 63.5 70.95

训练细节

TODO

Licence

本模型采用MIT licence.