|
--- |
|
tags: |
|
- mteb |
|
model-index: |
|
- name: stella-large-zh |
|
results: |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/AFQMC |
|
name: MTEB AFQMC |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 51.61327712288466 |
|
- type: cos_sim_spearman |
|
value: 54.48753880097122 |
|
- type: euclidean_pearson |
|
value: 52.68387289931342 |
|
- type: euclidean_spearman |
|
value: 54.48753879487172 |
|
- type: manhattan_pearson |
|
value: 52.635406372350026 |
|
- type: manhattan_spearman |
|
value: 54.447390526317044 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/ATEC |
|
name: MTEB ATEC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 53.39178036427897 |
|
- type: cos_sim_spearman |
|
value: 54.450028472876134 |
|
- type: euclidean_pearson |
|
value: 56.87300033777842 |
|
- type: euclidean_spearman |
|
value: 54.45002622056799 |
|
- type: manhattan_pearson |
|
value: 56.84326996138951 |
|
- type: manhattan_spearman |
|
value: 54.433880144849375 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (zh) |
|
config: zh |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 40.574000000000005 |
|
- type: f1 |
|
value: 38.87775700245793 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/BQ |
|
name: MTEB BQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 60.80957921870066 |
|
- type: cos_sim_spearman |
|
value: 62.37707350882237 |
|
- type: euclidean_pearson |
|
value: 61.29032932843765 |
|
- type: euclidean_spearman |
|
value: 62.37707350713817 |
|
- type: manhattan_pearson |
|
value: 61.23028102541801 |
|
- type: manhattan_spearman |
|
value: 62.31280056582247 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/CLSClusteringP2P |
|
name: MTEB CLSClusteringP2P |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 40.27066616318565 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/CLSClusteringS2S |
|
name: MTEB CLSClusteringS2S |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 37.503323644484716 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv1-reranking |
|
name: MTEB CMedQAv1 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 84.69295191328456 |
|
- type: mrr |
|
value: 87.08992063492063 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv2-reranking |
|
name: MTEB CMedQAv2 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 85.22650690364465 |
|
- type: mrr |
|
value: 87.72158730158729 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/CmedqaRetrieval |
|
name: MTEB CmedqaRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.54 |
|
- type: map_at_10 |
|
value: 35.591 |
|
- type: map_at_100 |
|
value: 37.549 |
|
- type: map_at_1000 |
|
value: 37.663000000000004 |
|
- type: map_at_3 |
|
value: 31.405 |
|
- type: map_at_5 |
|
value: 33.792 |
|
- type: mrr_at_1 |
|
value: 36.359 |
|
- type: mrr_at_10 |
|
value: 44.624 |
|
- type: mrr_at_100 |
|
value: 45.660000000000004 |
|
- type: mrr_at_1000 |
|
value: 45.707 |
|
- type: mrr_at_3 |
|
value: 42.002 |
|
- type: mrr_at_5 |
|
value: 43.535000000000004 |
|
- type: ndcg_at_1 |
|
value: 36.359 |
|
- type: ndcg_at_10 |
|
value: 42.28 |
|
- type: ndcg_at_100 |
|
value: 49.997 |
|
- type: ndcg_at_1000 |
|
value: 51.966 |
|
- type: ndcg_at_3 |
|
value: 36.851 |
|
- type: ndcg_at_5 |
|
value: 39.249 |
|
- type: precision_at_1 |
|
value: 36.359 |
|
- type: precision_at_10 |
|
value: 9.542 |
|
- type: precision_at_100 |
|
value: 1.582 |
|
- type: precision_at_1000 |
|
value: 0.183 |
|
- type: precision_at_3 |
|
value: 20.913999999999998 |
|
- type: precision_at_5 |
|
value: 15.404000000000002 |
|
- type: recall_at_1 |
|
value: 23.54 |
|
- type: recall_at_10 |
|
value: 53.005 |
|
- type: recall_at_100 |
|
value: 85.085 |
|
- type: recall_at_1000 |
|
value: 98.21 |
|
- type: recall_at_3 |
|
value: 36.944 |
|
- type: recall_at_5 |
|
value: 44.137 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: C-MTEB/CMNLI |
|
name: MTEB Cmnli |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 76.16355983162958 |
|
- type: cos_sim_ap |
|
value: 85.14228023901842 |
|
- type: cos_sim_f1 |
|
value: 77.86752827140549 |
|
- type: cos_sim_precision |
|
value: 72.18450479233228 |
|
- type: cos_sim_recall |
|
value: 84.5218611176058 |
|
- type: dot_accuracy |
|
value: 76.16355983162958 |
|
- type: dot_ap |
|
value: 85.16266644596179 |
|
- type: dot_f1 |
|
value: 77.86752827140549 |
|
- type: dot_precision |
|
value: 72.18450479233228 |
|
- type: dot_recall |
|
value: 84.5218611176058 |
|
- type: euclidean_accuracy |
|
value: 76.16355983162958 |
|
- type: euclidean_ap |
|
value: 85.14227717790371 |
|
- type: euclidean_f1 |
|
value: 77.86752827140549 |
|
- type: euclidean_precision |
|
value: 72.18450479233228 |
|
- type: euclidean_recall |
|
value: 84.5218611176058 |
|
- type: manhattan_accuracy |
|
value: 75.99518941671678 |
|
- type: manhattan_ap |
|
value: 85.10764940972825 |
|
- type: manhattan_f1 |
|
value: 77.80804694048618 |
|
- type: manhattan_precision |
|
value: 70.49553825707233 |
|
- type: manhattan_recall |
|
value: 86.81318681318682 |
|
- type: max_accuracy |
|
value: 76.16355983162958 |
|
- type: max_ap |
|
value: 85.16266644596179 |
|
- type: max_f1 |
|
value: 77.86752827140549 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/CovidRetrieval |
|
name: MTEB CovidRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 73.762 |
|
- type: map_at_10 |
|
value: 81.76299999999999 |
|
- type: map_at_100 |
|
value: 81.974 |
|
- type: map_at_1000 |
|
value: 81.977 |
|
- type: map_at_3 |
|
value: 80.23400000000001 |
|
- type: map_at_5 |
|
value: 81.189 |
|
- type: mrr_at_1 |
|
value: 74.18299999999999 |
|
- type: mrr_at_10 |
|
value: 81.792 |
|
- type: mrr_at_100 |
|
value: 81.994 |
|
- type: mrr_at_1000 |
|
value: 81.997 |
|
- type: mrr_at_3 |
|
value: 80.277 |
|
- type: mrr_at_5 |
|
value: 81.221 |
|
- type: ndcg_at_1 |
|
value: 74.078 |
|
- type: ndcg_at_10 |
|
value: 85.195 |
|
- type: ndcg_at_100 |
|
value: 86.041 |
|
- type: ndcg_at_1000 |
|
value: 86.111 |
|
- type: ndcg_at_3 |
|
value: 82.171 |
|
- type: ndcg_at_5 |
|
value: 83.90100000000001 |
|
- type: precision_at_1 |
|
value: 74.078 |
|
- type: precision_at_10 |
|
value: 9.684 |
|
- type: precision_at_100 |
|
value: 1.004 |
|
- type: precision_at_1000 |
|
value: 0.101 |
|
- type: precision_at_3 |
|
value: 29.470000000000002 |
|
- type: precision_at_5 |
|
value: 18.567 |
|
- type: recall_at_1 |
|
value: 73.762 |
|
- type: recall_at_10 |
|
value: 95.785 |
|
- type: recall_at_100 |
|
value: 99.368 |
|
- type: recall_at_1000 |
|
value: 99.895 |
|
- type: recall_at_3 |
|
value: 87.724 |
|
- type: recall_at_5 |
|
value: 91.93900000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/DuRetrieval |
|
name: MTEB DuRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.911 |
|
- type: map_at_10 |
|
value: 80.656 |
|
- type: map_at_100 |
|
value: 83.446 |
|
- type: map_at_1000 |
|
value: 83.485 |
|
- type: map_at_3 |
|
value: 55.998000000000005 |
|
- type: map_at_5 |
|
value: 70.577 |
|
- type: mrr_at_1 |
|
value: 90.14999999999999 |
|
- type: mrr_at_10 |
|
value: 93.35900000000001 |
|
- type: mrr_at_100 |
|
value: 93.419 |
|
- type: mrr_at_1000 |
|
value: 93.423 |
|
- type: mrr_at_3 |
|
value: 93.133 |
|
- type: mrr_at_5 |
|
value: 93.26100000000001 |
|
- type: ndcg_at_1 |
|
value: 90.14999999999999 |
|
- type: ndcg_at_10 |
|
value: 87.806 |
|
- type: ndcg_at_100 |
|
value: 90.4 |
|
- type: ndcg_at_1000 |
|
value: 90.776 |
|
- type: ndcg_at_3 |
|
value: 86.866 |
|
- type: ndcg_at_5 |
|
value: 85.619 |
|
- type: precision_at_1 |
|
value: 90.14999999999999 |
|
- type: precision_at_10 |
|
value: 42.045 |
|
- type: precision_at_100 |
|
value: 4.814 |
|
- type: precision_at_1000 |
|
value: 0.49 |
|
- type: precision_at_3 |
|
value: 78.0 |
|
- type: precision_at_5 |
|
value: 65.62 |
|
- type: recall_at_1 |
|
value: 25.911 |
|
- type: recall_at_10 |
|
value: 88.942 |
|
- type: recall_at_100 |
|
value: 97.56700000000001 |
|
- type: recall_at_1000 |
|
value: 99.62 |
|
- type: recall_at_3 |
|
value: 58.361 |
|
- type: recall_at_5 |
|
value: 75.126 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/EcomRetrieval |
|
name: MTEB EcomRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 46.2 |
|
- type: map_at_10 |
|
value: 56.309 |
|
- type: map_at_100 |
|
value: 56.977 |
|
- type: map_at_1000 |
|
value: 56.995 |
|
- type: map_at_3 |
|
value: 53.55 |
|
- type: map_at_5 |
|
value: 55.19 |
|
- type: mrr_at_1 |
|
value: 46.2 |
|
- type: mrr_at_10 |
|
value: 56.309 |
|
- type: mrr_at_100 |
|
value: 56.977 |
|
- type: mrr_at_1000 |
|
value: 56.995 |
|
- type: mrr_at_3 |
|
value: 53.55 |
|
- type: mrr_at_5 |
|
value: 55.19 |
|
- type: ndcg_at_1 |
|
value: 46.2 |
|
- type: ndcg_at_10 |
|
value: 61.656 |
|
- type: ndcg_at_100 |
|
value: 64.714 |
|
- type: ndcg_at_1000 |
|
value: 65.217 |
|
- type: ndcg_at_3 |
|
value: 56.022000000000006 |
|
- type: ndcg_at_5 |
|
value: 58.962 |
|
- type: precision_at_1 |
|
value: 46.2 |
|
- type: precision_at_10 |
|
value: 7.86 |
|
- type: precision_at_100 |
|
value: 0.9249999999999999 |
|
- type: precision_at_1000 |
|
value: 0.097 |
|
- type: precision_at_3 |
|
value: 21.067 |
|
- type: precision_at_5 |
|
value: 14.06 |
|
- type: recall_at_1 |
|
value: 46.2 |
|
- type: recall_at_10 |
|
value: 78.60000000000001 |
|
- type: recall_at_100 |
|
value: 92.5 |
|
- type: recall_at_1000 |
|
value: 96.5 |
|
- type: recall_at_3 |
|
value: 63.2 |
|
- type: recall_at_5 |
|
value: 70.3 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/IFlyTek-classification |
|
name: MTEB IFlyTek |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 47.03347441323585 |
|
- type: f1 |
|
value: 35.50895794566714 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/JDReview-classification |
|
name: MTEB JDReview |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 86.73545966228893 |
|
- type: ap |
|
value: 55.43694740493539 |
|
- type: f1 |
|
value: 81.47218440859787 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/LCQMC |
|
name: MTEB LCQMC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 70.49478085579923 |
|
- type: cos_sim_spearman |
|
value: 76.28442852235379 |
|
- type: euclidean_pearson |
|
value: 74.90910715249527 |
|
- type: euclidean_spearman |
|
value: 76.28443517178847 |
|
- type: manhattan_pearson |
|
value: 74.90744903779758 |
|
- type: manhattan_spearman |
|
value: 76.2886829916495 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/MMarcoRetrieval |
|
name: MTEB MMarcoRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 64.798 |
|
- type: map_at_10 |
|
value: 74.263 |
|
- type: map_at_100 |
|
value: 74.59 |
|
- type: map_at_1000 |
|
value: 74.601 |
|
- type: map_at_3 |
|
value: 72.382 |
|
- type: map_at_5 |
|
value: 73.59700000000001 |
|
- type: mrr_at_1 |
|
value: 67.049 |
|
- type: mrr_at_10 |
|
value: 74.86500000000001 |
|
- type: mrr_at_100 |
|
value: 75.155 |
|
- type: mrr_at_1000 |
|
value: 75.165 |
|
- type: mrr_at_3 |
|
value: 73.21600000000001 |
|
- type: mrr_at_5 |
|
value: 74.259 |
|
- type: ndcg_at_1 |
|
value: 67.049 |
|
- type: ndcg_at_10 |
|
value: 78.104 |
|
- type: ndcg_at_100 |
|
value: 79.56400000000001 |
|
- type: ndcg_at_1000 |
|
value: 79.85600000000001 |
|
- type: ndcg_at_3 |
|
value: 74.54499999999999 |
|
- type: ndcg_at_5 |
|
value: 76.587 |
|
- type: precision_at_1 |
|
value: 67.049 |
|
- type: precision_at_10 |
|
value: 9.493 |
|
- type: precision_at_100 |
|
value: 1.022 |
|
- type: precision_at_1000 |
|
value: 0.105 |
|
- type: precision_at_3 |
|
value: 28.189999999999998 |
|
- type: precision_at_5 |
|
value: 18.003 |
|
- type: recall_at_1 |
|
value: 64.798 |
|
- type: recall_at_10 |
|
value: 89.328 |
|
- type: recall_at_100 |
|
value: 95.916 |
|
- type: recall_at_1000 |
|
value: 98.223 |
|
- type: recall_at_3 |
|
value: 79.93599999999999 |
|
- type: recall_at_5 |
|
value: 84.789 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 64.01815736381977 |
|
- type: f1 |
|
value: 61.07806329750582 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 68.94754539340954 |
|
- type: f1 |
|
value: 68.76446930296682 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/MedicalRetrieval |
|
name: MTEB MedicalRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 50.1 |
|
- type: map_at_10 |
|
value: 56.406 |
|
- type: map_at_100 |
|
value: 56.958 |
|
- type: map_at_1000 |
|
value: 57.007 |
|
- type: map_at_3 |
|
value: 55.083000000000006 |
|
- type: map_at_5 |
|
value: 55.952999999999996 |
|
- type: mrr_at_1 |
|
value: 50.1 |
|
- type: mrr_at_10 |
|
value: 56.401999999999994 |
|
- type: mrr_at_100 |
|
value: 56.955 |
|
- type: mrr_at_1000 |
|
value: 57.004 |
|
- type: mrr_at_3 |
|
value: 55.05 |
|
- type: mrr_at_5 |
|
value: 55.95 |
|
- type: ndcg_at_1 |
|
value: 50.1 |
|
- type: ndcg_at_10 |
|
value: 59.384 |
|
- type: ndcg_at_100 |
|
value: 62.339 |
|
- type: ndcg_at_1000 |
|
value: 63.756 |
|
- type: ndcg_at_3 |
|
value: 56.657999999999994 |
|
- type: ndcg_at_5 |
|
value: 58.267 |
|
- type: precision_at_1 |
|
value: 50.1 |
|
- type: precision_at_10 |
|
value: 6.87 |
|
- type: precision_at_100 |
|
value: 0.832 |
|
- type: precision_at_1000 |
|
value: 0.095 |
|
- type: precision_at_3 |
|
value: 20.4 |
|
- type: precision_at_5 |
|
value: 13.04 |
|
- type: recall_at_1 |
|
value: 50.1 |
|
- type: recall_at_10 |
|
value: 68.7 |
|
- type: recall_at_100 |
|
value: 83.2 |
|
- type: recall_at_1000 |
|
value: 94.6 |
|
- type: recall_at_3 |
|
value: 61.199999999999996 |
|
- type: recall_at_5 |
|
value: 65.2 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/Mmarco-reranking |
|
name: MTEB MMarcoReranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 27.159122893681587 |
|
- type: mrr |
|
value: 25.659126984126985 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/MultilingualSentiment-classification |
|
name: MTEB MultilingualSentiment |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 73.02666666666667 |
|
- type: f1 |
|
value: 72.47691397067602 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: C-MTEB/OCNLI |
|
name: MTEB Ocnli |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 67.0817541959935 |
|
- type: cos_sim_ap |
|
value: 72.29133043915637 |
|
- type: cos_sim_f1 |
|
value: 72.71207689093188 |
|
- type: cos_sim_precision |
|
value: 60.16597510373444 |
|
- type: cos_sim_recall |
|
value: 91.86906019007391 |
|
- type: dot_accuracy |
|
value: 67.0817541959935 |
|
- type: dot_ap |
|
value: 72.29133043915637 |
|
- type: dot_f1 |
|
value: 72.71207689093188 |
|
- type: dot_precision |
|
value: 60.16597510373444 |
|
- type: dot_recall |
|
value: 91.86906019007391 |
|
- type: euclidean_accuracy |
|
value: 67.0817541959935 |
|
- type: euclidean_ap |
|
value: 72.29133043915637 |
|
- type: euclidean_f1 |
|
value: 72.71207689093188 |
|
- type: euclidean_precision |
|
value: 60.16597510373444 |
|
- type: euclidean_recall |
|
value: 91.86906019007391 |
|
- type: manhattan_accuracy |
|
value: 66.91932864103953 |
|
- type: manhattan_ap |
|
value: 72.20070509521395 |
|
- type: manhattan_f1 |
|
value: 72.52839713925118 |
|
- type: manhattan_precision |
|
value: 60.27972027972028 |
|
- type: manhattan_recall |
|
value: 91.02428722280888 |
|
- type: max_accuracy |
|
value: 67.0817541959935 |
|
- type: max_ap |
|
value: 72.29133043915637 |
|
- type: max_f1 |
|
value: 72.71207689093188 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/OnlineShopping-classification |
|
name: MTEB OnlineShopping |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 90.75000000000001 |
|
- type: ap |
|
value: 87.99706544930007 |
|
- type: f1 |
|
value: 90.72973221476978 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/PAWSX |
|
name: MTEB PAWSX |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 33.57372874898899 |
|
- type: cos_sim_spearman |
|
value: 37.9718472605281 |
|
- type: euclidean_pearson |
|
value: 38.52264008741102 |
|
- type: euclidean_spearman |
|
value: 37.97184654854654 |
|
- type: manhattan_pearson |
|
value: 38.50412571398273 |
|
- type: manhattan_spearman |
|
value: 37.98038173979437 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/QBQTC |
|
name: MTEB QBQTC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 37.510457667606225 |
|
- type: cos_sim_spearman |
|
value: 37.83522430820119 |
|
- type: euclidean_pearson |
|
value: 36.65815519443564 |
|
- type: euclidean_spearman |
|
value: 37.83519816393499 |
|
- type: manhattan_pearson |
|
value: 36.66835898210608 |
|
- type: manhattan_spearman |
|
value: 37.85390202705368 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (zh) |
|
config: zh |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 66.9953337569138 |
|
- type: cos_sim_spearman |
|
value: 67.27632129468024 |
|
- type: euclidean_pearson |
|
value: 65.83716645437758 |
|
- type: euclidean_spearman |
|
value: 67.27632129468024 |
|
- type: manhattan_pearson |
|
value: 65.81209103940279 |
|
- type: manhattan_spearman |
|
value: 67.26678679870099 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/STSB |
|
name: MTEB STSB |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 75.73719311549382 |
|
- type: cos_sim_spearman |
|
value: 75.71173848950517 |
|
- type: euclidean_pearson |
|
value: 75.23070020894484 |
|
- type: euclidean_spearman |
|
value: 75.71173839940812 |
|
- type: manhattan_pearson |
|
value: 75.23517292603057 |
|
- type: manhattan_spearman |
|
value: 75.74250916645184 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/T2Reranking |
|
name: MTEB T2Reranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 66.8596523608508 |
|
- type: mrr |
|
value: 76.9288884590171 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/T2Retrieval |
|
name: MTEB T2Retrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.618000000000002 |
|
- type: map_at_10 |
|
value: 74.884 |
|
- type: map_at_100 |
|
value: 78.65299999999999 |
|
- type: map_at_1000 |
|
value: 78.724 |
|
- type: map_at_3 |
|
value: 52.507999999999996 |
|
- type: map_at_5 |
|
value: 64.52799999999999 |
|
- type: mrr_at_1 |
|
value: 88.453 |
|
- type: mrr_at_10 |
|
value: 91.157 |
|
- type: mrr_at_100 |
|
value: 91.263 |
|
- type: mrr_at_1000 |
|
value: 91.268 |
|
- type: mrr_at_3 |
|
value: 90.672 |
|
- type: mrr_at_5 |
|
value: 90.96499999999999 |
|
- type: ndcg_at_1 |
|
value: 88.453 |
|
- type: ndcg_at_10 |
|
value: 82.759 |
|
- type: ndcg_at_100 |
|
value: 86.709 |
|
- type: ndcg_at_1000 |
|
value: 87.41499999999999 |
|
- type: ndcg_at_3 |
|
value: 84.194 |
|
- type: ndcg_at_5 |
|
value: 82.645 |
|
- type: precision_at_1 |
|
value: 88.453 |
|
- type: precision_at_10 |
|
value: 41.369 |
|
- type: precision_at_100 |
|
value: 4.9910000000000005 |
|
- type: precision_at_1000 |
|
value: 0.515 |
|
- type: precision_at_3 |
|
value: 73.79400000000001 |
|
- type: precision_at_5 |
|
value: 61.80799999999999 |
|
- type: recall_at_1 |
|
value: 26.618000000000002 |
|
- type: recall_at_10 |
|
value: 81.772 |
|
- type: recall_at_100 |
|
value: 94.55 |
|
- type: recall_at_1000 |
|
value: 98.184 |
|
- type: recall_at_3 |
|
value: 54.26499999999999 |
|
- type: recall_at_5 |
|
value: 67.963 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/TNews-classification |
|
name: MTEB TNews |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 50.690000000000005 |
|
- type: f1 |
|
value: 48.77079213417325 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/ThuNewsClusteringP2P |
|
name: MTEB ThuNewsClusteringP2P |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 62.14566804144758 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/ThuNewsClusteringS2S |
|
name: MTEB ThuNewsClusteringS2S |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 54.66890415410679 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/VideoRetrieval |
|
name: MTEB VideoRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 55.900000000000006 |
|
- type: map_at_10 |
|
value: 66.188 |
|
- type: map_at_100 |
|
value: 66.67699999999999 |
|
- type: map_at_1000 |
|
value: 66.691 |
|
- type: map_at_3 |
|
value: 64.017 |
|
- type: map_at_5 |
|
value: 65.462 |
|
- type: mrr_at_1 |
|
value: 55.800000000000004 |
|
- type: mrr_at_10 |
|
value: 66.13799999999999 |
|
- type: mrr_at_100 |
|
value: 66.62700000000001 |
|
- type: mrr_at_1000 |
|
value: 66.64099999999999 |
|
- type: mrr_at_3 |
|
value: 63.967 |
|
- type: mrr_at_5 |
|
value: 65.412 |
|
- type: ndcg_at_1 |
|
value: 55.900000000000006 |
|
- type: ndcg_at_10 |
|
value: 70.961 |
|
- type: ndcg_at_100 |
|
value: 73.22 |
|
- type: ndcg_at_1000 |
|
value: 73.583 |
|
- type: ndcg_at_3 |
|
value: 66.61 |
|
- type: ndcg_at_5 |
|
value: 69.18900000000001 |
|
- type: precision_at_1 |
|
value: 55.900000000000006 |
|
- type: precision_at_10 |
|
value: 8.58 |
|
- type: precision_at_100 |
|
value: 0.9610000000000001 |
|
- type: precision_at_1000 |
|
value: 0.099 |
|
- type: precision_at_3 |
|
value: 24.7 |
|
- type: precision_at_5 |
|
value: 16.06 |
|
- type: recall_at_1 |
|
value: 55.900000000000006 |
|
- type: recall_at_10 |
|
value: 85.8 |
|
- type: recall_at_100 |
|
value: 96.1 |
|
- type: recall_at_1000 |
|
value: 98.9 |
|
- type: recall_at_3 |
|
value: 74.1 |
|
- type: recall_at_5 |
|
value: 80.30000000000001 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/waimai-classification |
|
name: MTEB Waimai |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 86.77 |
|
- type: ap |
|
value: 70.21134107638184 |
|
- type: f1 |
|
value: 85.22521777795022 |
|
--- |
|
|
|
**新闻 | News** |
|
|
|
**[2024-04-06]** 开源[puff](https://huggingface.co/infgrad/puff-base-v1)系列模型，**专门针对检索和语义匹配任务，更多的考虑泛化性和私有通用测试集效果，向量维度可变，中英双语**。 |
|
|
|
**[2024-02-27]** 开源stella-mrl-large-zh-v3.5-1792d模型，支持**向量可变维度**。 |
|
|
|
**[2024-02-17]** 开源stella v3系列、dialogue编码模型和相关训练数据。 |
|
|
|
**[2023-10-19]** 开源stella-base-en-v2 使用简单，**不需要任何前缀文本**。 |
|
|
|
**[2023-10-12]** 开源stella-base-zh-v2和stella-large-zh-v2, 效果更好且使用简单，**不需要任何前缀文本**。 |
|
|
|
**[2023-09-11]** 开源stella-base-zh和stella-large-zh |
|
|
|
欢迎去[本人主页](https://huggingface.co/infgrad)查看最新模型，并提出您的宝贵意见！ |
|
|
|
|
|
## stella model |
|
|
|
|
|
stella是一个通用的文本编码模型，主要有以下模型： |
|
|
|
| Model Name | Model Size (GB) | Dimension | Sequence Length | Language | Need instruction for retrieval? | |
|
|:------------------:|:---------------:|:---------:|:---------------:|:--------:|:-------------------------------:| |
|
| stella-base-en-v2 | 0.2 | 768 | 512 | English | No | |
|
| stella-large-zh-v2 | 0.65 | 1024 | 1024 | Chinese | No | |
|
| stella-base-zh-v2 | 0.2 | 768 | 1024 | Chinese | No | |
|
| stella-large-zh | 0.65 | 1024 | 1024 | Chinese | Yes | |
|
| stella-base-zh | 0.2 | 768 | 1024 | Chinese | Yes | |
|
|
|
完整的训练思路和训练过程已记录在[博客1](https://zhuanlan.zhihu.com/p/655322183)和[博客2](https://zhuanlan.zhihu.com/p/662209559)，欢迎阅读讨论。 |
|
|
|
**训练数据：** |
|
|
|
1. 开源数据(wudao_base_200GB[1]、m3e[2]和simclue[3])，着重挑选了长度大于512的文本 |
|
2. 在通用语料库上使用LLM构造一批(question, paragraph)和(sentence, paragraph)数据 |
|
|
|
**训练方法：** |
|
|
|
1. 对比学习损失函数 |
|
2. 带有难负例的对比学习损失函数(分别基于bm25和vector构造了难负例) |
|
3. EWC(Elastic Weights Consolidation)[4] |
|
4. cosent loss[5] |
|
5. 每一种类型的数据一个迭代器，分别计算loss进行更新 |
|
|
|
stella-v2在stella模型的基础上，使用了更多的训练数据，同时知识蒸馏等方法去除了前置的instruction( |
|
比如piccolo的`查询:`, `结果:`, e5的`query:`和`passage:`)。 |
|
|
|
**初始权重：**\ |
|
stella-base-zh和stella-large-zh分别以piccolo-base-zh[6]和piccolo-large-zh作为基础模型，512-1024的position |
|
embedding使用层次分解位置编码[7]进行初始化。\ |
|
感谢商汤科技研究院开源的[piccolo系列模型](https://huggingface.co/sensenova)。 |
|
|
|
stella is a general-purpose text encoder, which mainly includes the following models: |
|
|
|
| Model Name | Model Size (GB) | Dimension | Sequence Length | Language | Need instruction for retrieval? | |
|
|:------------------:|:---------------:|:---------:|:---------------:|:--------:|:-------------------------------:| |
|
| stella-base-en-v2 | 0.2 | 768 | 512 | English | No | |
|
| stella-large-zh-v2 | 0.65 | 1024 | 1024 | Chinese | No | |
|
| stella-base-zh-v2 | 0.2 | 768 | 1024 | Chinese | No | |
|
| stella-large-zh | 0.65 | 1024 | 1024 | Chinese | Yes | |
|
| stella-base-zh | 0.2 | 768 | 1024 | Chinese | Yes | |
|
|
|
The training data mainly includes: |
|
|
|
1. Open-source training data (wudao_base_200GB, m3e, and simclue), with a focus on selecting texts with lengths greater |
|
than 512. |
|
2. A batch of (question, paragraph) and (sentence, paragraph) data constructed on a general corpus using LLM. |
|
|
|
The loss functions mainly include: |
|
|
|
1. Contrastive learning loss function |
|
2. Contrastive learning loss function with hard negative examples (based on bm25 and vector hard negatives) |
|
3. EWC (Elastic Weights Consolidation) |
|
4. cosent loss |
|
|
|
Model weight initialization:\ |
|
stella-base-zh and stella-large-zh use piccolo-base-zh and piccolo-large-zh as the base models, respectively, and the |
|
512-1024 position embedding uses the initialization strategy of hierarchical decomposed position encoding. |
|
|
|
Training strategy:\ |
|
One iterator for each type of data, separately calculating the loss. |
|
|
|
Based on stella models, stella-v2 use more training data and remove instruction by Knowledge Distillation. |
|
|
|
## Metric |
|
|
|
#### C-MTEB leaderboard (Chinese) |
|
|
|
| Model Name | Model Size (GB) | Dimension | Sequence Length | Average (35) | Classification (9) | Clustering (4) | Pair Classification (2) | Reranking (4) | Retrieval (8) | STS (8) | |
|
|:------------------:|:---------------:|:---------:|:---------------:|:------------:|:------------------:|:--------------:|:-----------------------:|:-------------:|:-------------:|:-------:| |
|
| stella-large-zh-v2 | 0.65 | 1024 | 1024 | 65.13 | 69.05 | 49.16 | 82.68 | 66.41 | 70.14 | 58.66 | |
|
| stella-base-zh-v2 | 0.2 | 768 | 1024 | 64.36 | 68.29 | 49.4 | 79.95 | 66.1 | 70.08 | 56.92 | |
|
| stella-large-zh | 0.65 | 1024 | 1024 | 64.54 | 67.62 | 48.65 | 78.72 | 65.98 | 71.02 | 58.3 | |
|
| stella-base-zh | 0.2 | 768 | 1024 | 64.16 | 67.77 | 48.7 | 76.09 | 66.95 | 71.07 | 56.54 | |
|
|
|
#### MTEB leaderboard (English) |
|
|
|
| Model Name | Model Size (GB) | Dimension | Sequence Length | Average (56) | Classification (12) | Clustering (11) | Pair Classification (3) | Reranking (4) | Retrieval (15) | STS (10) | Summarization (1) | |
|
|:-----------------:|:---------------:|:---------:|:---------------:|:------------:|:-------------------:|:---------------:|:-----------------------:|:-------------:|:--------------:|:--------:|:------------------:| |
|
| stella-base-en-v2 | 0.2 | 768 | 512 | 62.61 | 75.28 | 44.9 | 86.45 | 58.77 | 50.1 | 83.02 | 32.52 | |
|
|
|
#### Reproduce our results |
|
|
|
**C-MTEB:** |
|
|
|
```python |
|
import torch |
|
import numpy as np |
|
from typing import List |
|
from mteb import MTEB |
|
from sentence_transformers import SentenceTransformer |
|
|
|
|
|
class FastTextEncoder(): |
|
def __init__(self, model_name): |
|
self.model = SentenceTransformer(model_name).cuda().half().eval() |
|
self.model.max_seq_length = 512 |
|
|
|
def encode( |
|
self, |
|
input_texts: List[str], |
|
*args, |
|
**kwargs |
|
): |
|
new_sens = list(set(input_texts)) |
|
new_sens.sort(key=lambda x: len(x), reverse=True) |
|
vecs = self.model.encode( |
|
new_sens, normalize_embeddings=True, convert_to_numpy=True, batch_size=256 |
|
).astype(np.float32) |
|
sen2arrid = {sen: idx for idx, sen in enumerate(new_sens)} |
|
vecs = vecs[[sen2arrid[sen] for sen in input_texts]] |
|
torch.cuda.empty_cache() |
|
return vecs |
|
|
|
|
|
if __name__ == '__main__': |
|
model_name = "infgrad/stella-base-zh-v2" |
|
output_folder = "zh_mteb_results/stella-base-zh-v2" |
|
task_names = [t.description["name"] for t in MTEB(task_langs=['zh', 'zh-CN']).tasks] |
|
model = FastTextEncoder(model_name) |
|
for task in task_names: |
|
MTEB(tasks=[task], task_langs=['zh', 'zh-CN']).run(model, output_folder=output_folder) |
|
|
|
``` |
|
|
|
**MTEB:** |
|
|
|
You can use official script to reproduce our result. [scripts/run_mteb_english.py](https://github.com/embeddings-benchmark/mteb/blob/main/scripts/run_mteb_english.py) |
|
|
|
#### Evaluation for long text |
|
|
|
经过实际观察发现，C-MTEB的评测数据长度基本都是小于512的， |
|
更致命的是那些长度大于512的文本，其重点都在前半部分 |
|
这里以CMRC2018的数据为例说明这个问题： |
|
|
|
``` |
|
question: 《无双大蛇z》是谁旗下ω-force开发的动作游戏？ |
|
|
|
passage：《无双大蛇z》是光荣旗下ω-force开发的动作游戏，于2009年3月12日登陆索尼playstation3，并于2009年11月27日推...... |
|
``` |
|
|
|
passage长度为800多，大于512，但是对于这个question而言只需要前面40个字就足以检索，多的内容对于模型而言是一种噪声，反而降低了效果。\ |
|
简言之，现有数据集的2个问题：\ |
|
1）长度大于512的过少\ |
|
2）即便大于512，对于检索而言也只需要前512的文本内容\ |
|
导致**无法准确评估模型的长文本编码能力。** |
|
|
|
为了解决这个问题，搜集了相关开源数据并使用规则进行过滤，最终整理了6份长文本测试集,他们分别是： |
|
|
|
- CMRC2018，通用百科 |
|
- CAIL，法律阅读理解 |
|
- DRCD，繁体百科，已转简体 |
|
- Military，军工问答 |
|
- Squad，英文阅读理解，已转中文 |
|
- Multifieldqa_zh，清华的大模型长文本理解能力评测数据[9] |
|
|
|
处理规则是选取答案在512长度之后的文本，短的测试数据会欠采样一下，长短文本占比约为1:2，所以模型既得理解短文本也得理解长文本。 |
|
除了Military数据集，我们提供了其他5个测试数据的下载地址：https://drive.google.com/file/d/1WC6EWaCbVgz-vPMDFH4TwAMkLyh5WNcN/view?usp=sharing |
|
|
|
评测指标为Recall@5, 结果如下： |
|
|
|
| Dataset | piccolo-base-zh | piccolo-large-zh | bge-base-zh | bge-large-zh | stella-base-zh | stella-large-zh | |
|
|:---------------:|:---------------:|:----------------:|:-----------:|:------------:|:--------------:|:---------------:| |
|
| CMRC2018 | 94.34 | 93.82 | 91.56 | 93.12 | 96.08 | 95.56 | |
|
| CAIL | 28.04 | 33.64 | 31.22 | 33.94 | 34.62 | 37.18 | |
|
| DRCD | 78.25 | 77.9 | 78.34 | 80.26 | 86.14 | 84.58 | |
|
| Military | 76.61 | 73.06 | 75.65 | 75.81 | 83.71 | 80.48 | |
|
| Squad | 91.21 | 86.61 | 87.87 | 90.38 | 93.31 | 91.21 | |
|
| Multifieldqa_zh | 81.41 | 83.92 | 83.92 | 83.42 | 79.9 | 80.4 | |
|
| **Average** | 74.98 | 74.83 | 74.76 | 76.15 | **78.96** | **78.24** | |
|
|
|
**注意：** 因为长文本评测数据数量稀少，所以构造时也使用了train部分，如果自行评测，请注意模型的训练数据以免数据泄露。 |
|
|
|
## Usage |
|
|
|
#### stella 中文系列模型 |
|
|
|
stella-base-zh 和 stella-large-zh: 本模型是在piccolo基础上训练的，因此**用法和piccolo完全一致** |
|
，即在检索重排任务上给query和passage加上`查询: `和`结果: `。对于短短匹配不需要做任何操作。 |
|
|
|
stella-base-zh-v2 和 stella-large-zh-v2: 本模型使用简单，**任何使用场景中都不需要加前缀文本**。 |
|
|
|
stella中文系列模型均使用mean pooling做为文本向量。 |
|
|
|
在sentence-transformer库中的使用方法： |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
|
|
sentences = ["数据1", "数据2"] |
|
model = SentenceTransformer('infgrad/stella-base-zh-v2') |
|
print(model.max_seq_length) |
|
embeddings_1 = model.encode(sentences, normalize_embeddings=True) |
|
embeddings_2 = model.encode(sentences, normalize_embeddings=True) |
|
similarity = embeddings_1 @ embeddings_2.T |
|
print(similarity) |
|
``` |
|
|
|
直接使用transformers库： |
|
|
|
```python |
|
from transformers import AutoModel, AutoTokenizer |
|
from sklearn.preprocessing import normalize |
|
|
|
model = AutoModel.from_pretrained('infgrad/stella-base-zh-v2') |
|
tokenizer = AutoTokenizer.from_pretrained('infgrad/stella-base-zh-v2') |
|
sentences = ["数据1", "数据ABCDEFGH"] |
|
batch_data = tokenizer( |
|
batch_text_or_text_pairs=sentences, |
|
padding="longest", |
|
return_tensors="pt", |
|
max_length=1024, |
|
truncation=True, |
|
) |
|
attention_mask = batch_data["attention_mask"] |
|
model_output = model(**batch_data) |
|
last_hidden = model_output.last_hidden_state.masked_fill(~attention_mask[..., None].bool(), 0.0) |
|
vectors = last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] |
|
vectors = normalize(vectors, norm="l2", axis=1, ) |
|
print(vectors.shape) # 2,768 |
|
``` |
|
|
|
#### stella models for English |
|
|
|
**Using Sentence-Transformers:** |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
|
|
sentences = ["one car come", "one car go"] |
|
model = SentenceTransformer('infgrad/stella-base-en-v2') |
|
print(model.max_seq_length) |
|
embeddings_1 = model.encode(sentences, normalize_embeddings=True) |
|
embeddings_2 = model.encode(sentences, normalize_embeddings=True) |
|
similarity = embeddings_1 @ embeddings_2.T |
|
print(similarity) |
|
``` |
|
|
|
**Using HuggingFace Transformers:** |
|
|
|
```python |
|
from transformers import AutoModel, AutoTokenizer |
|
from sklearn.preprocessing import normalize |
|
|
|
model = AutoModel.from_pretrained('infgrad/stella-base-en-v2') |
|
tokenizer = AutoTokenizer.from_pretrained('infgrad/stella-base-en-v2') |
|
sentences = ["one car come", "one car go"] |
|
batch_data = tokenizer( |
|
batch_text_or_text_pairs=sentences, |
|
padding="longest", |
|
return_tensors="pt", |
|
max_length=512, |
|
truncation=True, |
|
) |
|
attention_mask = batch_data["attention_mask"] |
|
model_output = model(**batch_data) |
|
last_hidden = model_output.last_hidden_state.masked_fill(~attention_mask[..., None].bool(), 0.0) |
|
vectors = last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] |
|
vectors = normalize(vectors, norm="l2", axis=1, ) |
|
print(vectors.shape) # 2,768 |
|
``` |
|
|
|
## Training Detail |
|
|
|
**硬件：** 单卡A100-80GB |
|
|
|
**环境：** torch1.13.*; transformers-trainer + deepspeed + gradient-checkpointing |
|
|
|
**学习率：** 1e-6 |
|
|
|
**batch_size：** base模型为1024，额外增加20%的难负例；large模型为768，额外增加20%的难负例 |
|
|
|
**数据量：** 第一版模型约100万，其中用LLM构造的数据约有200K. LLM模型大小为13b。v2系列模型到了2000万训练数据。 |
|
|
|
## ToDoList |
|
|
|
**评测的稳定性：** |
|
评测过程中发现Clustering任务会和官方的结果不一致，大约有±0.0x的小差距，原因是聚类代码没有设置random_seed，差距可以忽略不计，不影响评测结论。 |
|
|
|
**更高质量的长文本训练和测试数据：** 训练数据多是用13b模型构造的，肯定会存在噪声。 |
|
测试数据基本都是从mrc数据整理来的，所以问题都是factoid类型，不符合真实分布。 |
|
|
|
**OOD的性能：** 虽然近期出现了很多向量编码模型，但是对于不是那么通用的domain，这一众模型包括stella、openai和cohere, |
|
它们的效果均比不上BM25。 |
|
|
|
## Reference |
|
|
|
1. https://www.scidb.cn/en/detail?dataSetId=c6a3fe684227415a9db8e21bac4a15ab |
|
2. https://github.com/wangyuxinwhy/uniem |
|
3. https://github.com/CLUEbenchmark/SimCLUE |
|
4. https://arxiv.org/abs/1612.00796 |
|
5. https://kexue.fm/archives/8847 |
|
6. https://huggingface.co/sensenova/piccolo-base-zh |
|
7. https://kexue.fm/archives/7947 |
|
8. https://github.com/FlagOpen/FlagEmbedding |
|
9. https://github.com/THUDM/LongBench |
|
|
|
|
|
|
|
|
|
|
|
|
