--- language: - en license: mit model-index: - name: Zeta-Alpha-E5-Mistral results: - dataset: config: en name: MTEB AmazonCounterfactualClassification (en) revision: e8379541af4e31359cca9fbcf4b00f2671dba205 split: test type: mteb/amazon_counterfactual metrics: - type: accuracy value: 77.76119402985074 - type: ap value: 39.97673988468886 - type: ap_weighted value: 39.97673988468886 - type: f1 value: 71.23171737695898 - type: f1_weighted value: 79.55230307558237 - type: main_score value: 77.76119402985074 task: type: Classification - dataset: config: default name: MTEB AmazonPolarityClassification revision: e2d317d38cd51312af73b3d32a06d1a08b442046 split: test type: mteb/amazon_polarity metrics: - type: accuracy value: 96.61810000000001 - type: ap value: 94.99559013902017 - type: ap_weighted value: 94.99559013902017 - type: f1 value: 96.61758649480731 - type: f1_weighted value: 96.61758649480731 - type: main_score value: 96.61810000000001 task: type: Classification - dataset: config: en name: MTEB AmazonReviewsClassification (en) revision: 1399c76144fd37290681b995c656ef9b2e06e26d split: test type: mteb/amazon_reviews_multi metrics: - type: accuracy value: 59.26199999999999 - type: f1 value: 56.32963321217333 - type: f1_weighted value: 56.32963321217333 - type: main_score value: 59.26199999999999 task: type: Classification - dataset: config: default name: MTEB ArguAna revision: c22ab2a51041ffd869aaddef7af8d8215647e41a split: test type: mteb/arguana metrics: - type: main_score value: 65.623 - type: map_at_1 value: 41.536 - type: map_at_10 value: 57.485 - type: map_at_100 value: 58.013000000000005 - type: map_at_1000 value: 58.013000000000005 - type: map_at_20 value: 57.957 - type: map_at_3 value: 53.284 - type: map_at_5 value: 55.837 - type: mrr_at_1 value: 42.17638691322902 - type: mrr_at_10 value: 57.7096175122492 - type: mrr_at_100 value: 58.23610809196743 - type: mrr_at_1000 value: 58.23673750573145 - type: mrr_at_20 value: 58.180348622747324 - type: mrr_at_3 value: 53.44950213371275 - type: mrr_at_5 value: 56.07396870554779 - type: nauc_map_at_1000_diff1 value: 14.098091070036958 - type: nauc_map_at_1000_max value: -16.568377380844108 - type: nauc_map_at_1000_std value: -22.105696179585834 - type: nauc_map_at_100_diff1 value: 14.096542453201625 - type: nauc_map_at_100_max value: -16.57054590195526 - type: nauc_map_at_100_std value: -22.1090324366772 - type: nauc_map_at_10_diff1 value: 13.840246695558884 - type: nauc_map_at_10_max value: -16.52098795923224 - type: nauc_map_at_10_std value: -22.074328710004032 - type: nauc_map_at_1_diff1 value: 17.117727049808984 - type: nauc_map_at_1_max value: -18.587242049712614 - type: nauc_map_at_1_std value: -22.454707653498595 - type: nauc_map_at_20_diff1 value: 14.068130846454585 - type: nauc_map_at_20_max value: -16.53942858114966 - type: nauc_map_at_20_std value: -22.10921004077996 - type: nauc_map_at_3_diff1 value: 14.596579595737097 - type: nauc_map_at_3_max value: -15.62887067464894 - type: nauc_map_at_3_std value: -22.09058102549274 - type: nauc_map_at_5_diff1 value: 13.798507062284514 - type: nauc_map_at_5_max value: -16.36834850771522 - type: nauc_map_at_5_std value: -21.984206595455134 - type: nauc_mrr_at_1000_diff1 value: 12.144909427474602 - type: nauc_mrr_at_1000_max value: -17.048787138426324 - type: nauc_mrr_at_1000_std value: -21.961966140898564 - type: nauc_mrr_at_100_diff1 value: 12.143403633612827 - type: nauc_mrr_at_100_max value: -17.050945262411012 - type: nauc_mrr_at_100_std value: -21.965305811191673 - type: nauc_mrr_at_10_diff1 value: 11.88548720648553 - type: nauc_mrr_at_10_max value: -16.996705857584736 - type: nauc_mrr_at_10_std value: -21.883645748542396 - type: nauc_mrr_at_1_diff1 value: 15.37682964765565 - type: nauc_mrr_at_1_max value: -17.989361001169087 - type: nauc_mrr_at_1_std value: -21.697830490637955 - type: nauc_mrr_at_20_diff1 value: 12.119044499779363 - type: nauc_mrr_at_20_max value: -17.018675761117027 - type: nauc_mrr_at_20_std value: -21.965554459307565 - type: nauc_mrr_at_3_diff1 value: 12.535001807278187 - type: nauc_mrr_at_3_max value: -16.38816957172248 - type: nauc_mrr_at_3_std value: -22.081293367465896 - type: nauc_mrr_at_5_diff1 value: 11.892111947679496 - type: nauc_mrr_at_5_max value: -16.79709351116846 - type: nauc_mrr_at_5_std value: -21.79512696140714 - type: nauc_ndcg_at_1000_diff1 value: 13.67006999549869 - type: nauc_ndcg_at_1000_max value: -16.236125687432107 - type: nauc_ndcg_at_1000_std value: -21.810131960233065 - type: nauc_ndcg_at_100_diff1 value: 13.637478389163462 - type: nauc_ndcg_at_100_max value: -16.28219720987127 - type: nauc_ndcg_at_100_std value: -21.880912370176876 - type: nauc_ndcg_at_10_diff1 value: 12.558591199280556 - type: nauc_ndcg_at_10_max value: -15.952826009827106 - type: nauc_ndcg_at_10_std value: -21.818643731025382 - type: nauc_ndcg_at_1_diff1 value: 17.117727049808984 - type: nauc_ndcg_at_1_max value: -18.587242049712614 - type: nauc_ndcg_at_1_std value: -22.454707653498595 - type: nauc_ndcg_at_20_diff1 value: 13.402986057386181 - type: nauc_ndcg_at_20_max value: -16.072631062968746 - type: nauc_ndcg_at_20_std value: -21.98468803430586 - type: nauc_ndcg_at_3_diff1 value: 14.059904782033348 - type: nauc_ndcg_at_3_max value: -14.433190101994514 - type: nauc_ndcg_at_3_std value: -21.990025270634135 - type: nauc_ndcg_at_5_diff1 value: 12.434165121057134 - type: nauc_ndcg_at_5_max value: -15.650774158031522 - type: nauc_ndcg_at_5_std value: -21.636716447934305 - type: nauc_precision_at_1000_diff1 value: 1.7151819945276745 - type: nauc_precision_at_1000_max value: 20.85546049013785 - type: nauc_precision_at_1000_std value: 77.3551884133584 - type: nauc_precision_at_100_diff1 value: -7.961881099019577 - type: nauc_precision_at_100_max value: -1.8225484680865736 - type: nauc_precision_at_100_std value: 35.484449109425384 - type: nauc_precision_at_10_diff1 value: 0.46638305609538855 - type: nauc_precision_at_10_max value: -11.023993018739485 - type: nauc_precision_at_10_std value: -19.111584616037852 - type: nauc_precision_at_1_diff1 value: 17.117727049808984 - type: nauc_precision_at_1_max value: -18.587242049712614 - type: nauc_precision_at_1_std value: -22.454707653498595 - type: nauc_precision_at_20_diff1 value: -1.0298881487766305 - type: nauc_precision_at_20_max value: -4.548017977674335 - type: nauc_precision_at_20_std value: -18.901496352112133 - type: nauc_precision_at_3_diff1 value: 12.350178962124566 - type: nauc_precision_at_3_max value: -10.271126387937858 - type: nauc_precision_at_3_std value: -21.655307623793433 - type: nauc_precision_at_5_diff1 value: 6.011571432832696 - type: nauc_precision_at_5_max value: -12.478026665421389 - type: nauc_precision_at_5_std value: -19.845124181363882 - type: nauc_recall_at_1000_diff1 value: 1.7151819945236155 - type: nauc_recall_at_1000_max value: 20.855460490135933 - type: nauc_recall_at_1000_std value: 77.35518841335626 - type: nauc_recall_at_100_diff1 value: -7.961881099020542 - type: nauc_recall_at_100_max value: -1.8225484680932273 - type: nauc_recall_at_100_std value: 35.48444910942399 - type: nauc_recall_at_10_diff1 value: 0.46638305609538805 - type: nauc_recall_at_10_max value: -11.023993018739322 - type: nauc_recall_at_10_std value: -19.11158461603798 - type: nauc_recall_at_1_diff1 value: 17.117727049808984 - type: nauc_recall_at_1_max value: -18.587242049712614 - type: nauc_recall_at_1_std value: -22.454707653498595 - type: nauc_recall_at_20_diff1 value: -1.029888148776229 - type: nauc_recall_at_20_max value: -4.548017977673906 - type: nauc_recall_at_20_std value: -18.901496352110804 - type: nauc_recall_at_3_diff1 value: 12.350178962124682 - type: nauc_recall_at_3_max value: -10.271126387937805 - type: nauc_recall_at_3_std value: -21.65530762379337 - type: nauc_recall_at_5_diff1 value: 6.0115714328326435 - type: nauc_recall_at_5_max value: -12.478026665421405 - type: nauc_recall_at_5_std value: -19.845124181363875 - type: ndcg_at_1 value: 41.536 - type: ndcg_at_10 value: 65.623 - type: ndcg_at_100 value: 67.63 - type: ndcg_at_1000 value: 67.64099999999999 - type: ndcg_at_20 value: 67.241 - type: ndcg_at_3 value: 57.048 - type: ndcg_at_5 value: 61.678999999999995 - type: precision_at_1 value: 41.536 - type: precision_at_10 value: 9.132 - type: precision_at_100 value: 0.996 - type: precision_at_1000 value: 0.1 - type: precision_at_20 value: 4.8759999999999994 - type: precision_at_3 value: 22.641 - type: precision_at_5 value: 15.845999999999998 - type: recall_at_1 value: 41.536 - type: recall_at_10 value: 91.323 - type: recall_at_100 value: 99.57300000000001 - type: recall_at_1000 value: 99.644 - type: recall_at_20 value: 97.51100000000001 - type: recall_at_3 value: 67.923 - type: recall_at_5 value: 79.232 task: type: Retrieval - dataset: config: default name: MTEB ArxivClusteringP2P revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d split: test type: mteb/arxiv-clustering-p2p metrics: - type: main_score value: 52.026635060007244 - type: v_measure value: 52.026635060007244 - type: v_measure_std value: 14.357137408692006 task: type: Clustering - dataset: config: default name: MTEB ArxivClusteringS2S revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 split: test type: mteb/arxiv-clustering-s2s metrics: - type: main_score value: 47.834914950269855 - type: v_measure value: 47.834914950269855 - type: v_measure_std value: 14.487028918517247 task: type: Clustering - dataset: config: default name: MTEB AskUbuntuDupQuestions revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 split: test type: mteb/askubuntudupquestions-reranking metrics: - type: main_score value: 64.5808745066313 - type: map value: 64.5808745066313 - type: mrr value: 77.56540473991997 - type: nAUC_map_diff1 value: 23.168800779252464 - type: nAUC_map_max value: 30.342203769599735 - type: nAUC_map_std value: 22.562701982176833 - type: nAUC_mrr_diff1 value: 27.79261544540621 - type: nAUC_mrr_max value: 43.302228243606045 - type: nAUC_mrr_std value: 26.432985515912673 task: type: Reranking - dataset: config: default name: MTEB BIOSSES revision: d3fb88f8f02e40887cd149695127462bbcf29b4a split: test type: mteb/biosses-sts metrics: - type: cosine_pearson value: 87.97778619889539 - type: cosine_spearman value: 86.44233109293758 - type: euclidean_pearson value: 86.6664224630525 - type: euclidean_spearman value: 86.44233109293758 - type: main_score value: 86.44233109293758 - type: manhattan_pearson value: 86.75174487553707 - type: manhattan_spearman value: 86.61402175201368 - type: pearson value: 87.97778619889539 - type: spearman value: 86.44233109293758 task: type: STS - dataset: config: default name: MTEB Banking77Classification revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 split: test type: mteb/banking77 metrics: - type: accuracy value: 83.1103896103896 - type: f1 value: 82.2932953112279 - type: f1_weighted value: 82.2932953112279 - type: main_score value: 83.1103896103896 task: type: Classification - dataset: config: default name: MTEB BiorxivClusteringP2P revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 split: test type: mteb/biorxiv-clustering-p2p metrics: - type: main_score value: 43.73746639290943 - type: v_measure value: 43.73746639290943 - type: v_measure_std value: 0.8808902310879784 task: type: Clustering - dataset: config: default name: MTEB BiorxivClusteringS2S revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 split: test type: mteb/biorxiv-clustering-s2s metrics: - type: main_score value: 40.73737737926463 - type: v_measure value: 40.73737737926463 - type: v_measure_std value: 0.6059982328960863 task: type: Clustering - dataset: config: default name: MTEB CQADupstackRetrieval revision: 46989137a86843e03a6195de44b09deda022eec7 split: test type: CQADupstackRetrieval_is_a_combined_dataset metrics: - type: main_score value: 48.852333333333334 - type: ndcg_at_10 value: 48.852333333333334 task: type: Retrieval - dataset: config: default name: MTEB ClimateFEVER revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 split: test type: mteb/climate-fever metrics: - type: main_score value: 42.047000000000004 - type: map_at_1 value: 18.269 - type: map_at_10 value: 31.691999999999997 - type: map_at_100 value: 33.841 - type: map_at_1000 value: 34.009 - type: map_at_20 value: 32.885999999999996 - type: map_at_3 value: 26.558999999999997 - type: map_at_5 value: 29.119 - type: mrr_at_1 value: 41.69381107491857 - type: mrr_at_10 value: 54.293185460937984 - type: mrr_at_100 value: 54.87161862744807 - type: mrr_at_1000 value: 54.88722882443645 - type: mrr_at_20 value: 54.685754380853844 - type: mrr_at_3 value: 51.172638436482146 - type: mrr_at_5 value: 53.12052117263853 - type: nauc_map_at_1000_diff1 value: 27.03807365621228 - type: nauc_map_at_1000_max value: 40.31079671403445 - type: nauc_map_at_1000_std value: 26.092423798773883 - type: nauc_map_at_100_diff1 value: 27.015360474734436 - type: nauc_map_at_100_max value: 40.28408194597505 - type: nauc_map_at_100_std value: 26.086029014261968 - type: nauc_map_at_10_diff1 value: 27.222731203652856 - type: nauc_map_at_10_max value: 40.01781109904128 - type: nauc_map_at_10_std value: 24.73681887890272 - type: nauc_map_at_1_diff1 value: 35.75107300356484 - type: nauc_map_at_1_max value: 40.2201153742901 - type: nauc_map_at_1_std value: 18.766249947929374 - type: nauc_map_at_20_diff1 value: 26.931804653893042 - type: nauc_map_at_20_max value: 40.21587995014608 - type: nauc_map_at_20_std value: 25.75452108695598 - type: nauc_map_at_3_diff1 value: 28.310387788680696 - type: nauc_map_at_3_max value: 39.285866053656385 - type: nauc_map_at_3_std value: 21.394962842915703 - type: nauc_map_at_5_diff1 value: 27.300839773785274 - type: nauc_map_at_5_max value: 39.3888708340898 - type: nauc_map_at_5_std value: 22.78299335246201 - type: nauc_mrr_at_1000_diff1 value: 26.569029993582287 - type: nauc_mrr_at_1000_max value: 38.05698386072128 - type: nauc_mrr_at_1000_std value: 27.12877875031529 - type: nauc_mrr_at_100_diff1 value: 26.56693868451222 - type: nauc_mrr_at_100_max value: 38.06321319344823 - type: nauc_mrr_at_100_std value: 27.14409997788537 - type: nauc_mrr_at_10_diff1 value: 26.52694223161396 - type: nauc_mrr_at_10_max value: 38.120563154705 - type: nauc_mrr_at_10_std value: 27.11337497751667 - type: nauc_mrr_at_1_diff1 value: 29.371725407886277 - type: nauc_mrr_at_1_max value: 35.7850341702808 - type: nauc_mrr_at_1_std value: 22.69810863765783 - type: nauc_mrr_at_20_diff1 value: 26.567897033309247 - type: nauc_mrr_at_20_max value: 38.17484491649562 - type: nauc_mrr_at_20_std value: 27.218678564296972 - type: nauc_mrr_at_3_diff1 value: 26.582727973322427 - type: nauc_mrr_at_3_max value: 37.8745721692282 - type: nauc_mrr_at_3_std value: 26.567749469034307 - type: nauc_mrr_at_5_diff1 value: 26.404958533442898 - type: nauc_mrr_at_5_max value: 37.86090955141593 - type: nauc_mrr_at_5_std value: 26.65816459603454 - type: nauc_ndcg_at_1000_diff1 value: 25.7228323702742 - type: nauc_ndcg_at_1000_max value: 41.024272689913296 - type: nauc_ndcg_at_1000_std value: 31.373617783353815 - type: nauc_ndcg_at_100_diff1 value: 25.467806967471812 - type: nauc_ndcg_at_100_max value: 40.68595692225817 - type: nauc_ndcg_at_100_std value: 31.327255356351774 - type: nauc_ndcg_at_10_diff1 value: 25.65771458118311 - type: nauc_ndcg_at_10_max value: 40.2959313004829 - type: nauc_ndcg_at_10_std value: 28.21103387387833 - type: nauc_ndcg_at_1_diff1 value: 29.371725407886277 - type: nauc_ndcg_at_1_max value: 35.7850341702808 - type: nauc_ndcg_at_1_std value: 22.69810863765783 - type: nauc_ndcg_at_20_diff1 value: 25.008107221444327 - type: nauc_ndcg_at_20_max value: 40.613619354979626 - type: nauc_ndcg_at_20_std value: 30.216191744111416 - type: nauc_ndcg_at_3_diff1 value: 25.85227194113396 - type: nauc_ndcg_at_3_max value: 38.32492256264965 - type: nauc_ndcg_at_3_std value: 23.735358525961033 - type: nauc_ndcg_at_5_diff1 value: 25.747409532466243 - type: nauc_ndcg_at_5_max value: 39.4993084566524 - type: nauc_ndcg_at_5_std value: 25.19771375383721 - type: nauc_precision_at_1000_diff1 value: -8.149028290279253 - type: nauc_precision_at_1000_max value: -3.196086649201077 - type: nauc_precision_at_1000_std value: 13.643701012139948 - type: nauc_precision_at_100_diff1 value: -1.892485292157653 - type: nauc_precision_at_100_max value: 7.7434454354621245 - type: nauc_precision_at_100_std value: 22.988854451791806 - type: nauc_precision_at_10_diff1 value: 6.150550804828545 - type: nauc_precision_at_10_max value: 22.501131175285906 - type: nauc_precision_at_10_std value: 27.39677272392596 - type: nauc_precision_at_1_diff1 value: 29.371725407886277 - type: nauc_precision_at_1_max value: 35.7850341702808 - type: nauc_precision_at_1_std value: 22.69810863765783 - type: nauc_precision_at_20_diff1 value: 2.283445965946842 - type: nauc_precision_at_20_max value: 18.59466543059599 - type: nauc_precision_at_20_std value: 29.0738299597803 - type: nauc_precision_at_3_diff1 value: 12.963867454979258 - type: nauc_precision_at_3_max value: 30.449562657056333 - type: nauc_precision_at_3_std value: 25.581976194336352 - type: nauc_precision_at_5_diff1 value: 8.512947940252289 - type: nauc_precision_at_5_max value: 26.12425424420038 - type: nauc_precision_at_5_std value: 24.877415885322808 - type: nauc_recall_at_1000_diff1 value: 17.151717317242028 - type: nauc_recall_at_1000_max value: 40.67913325938115 - type: nauc_recall_at_1000_std value: 49.54837910314142 - type: nauc_recall_at_100_diff1 value: 16.83432440063162 - type: nauc_recall_at_100_max value: 34.46952489534257 - type: nauc_recall_at_100_std value: 38.26853671426454 - type: nauc_recall_at_10_diff1 value: 19.50239551179883 - type: nauc_recall_at_10_max value: 35.74261290262663 - type: nauc_recall_at_10_std value: 28.630457514118163 - type: nauc_recall_at_1_diff1 value: 35.75107300356484 - type: nauc_recall_at_1_max value: 40.2201153742901 - type: nauc_recall_at_1_std value: 18.766249947929374 - type: nauc_recall_at_20_diff1 value: 16.723000685755707 - type: nauc_recall_at_20_max value: 35.272383093342405 - type: nauc_recall_at_20_std value: 32.934757635631335 - type: nauc_recall_at_3_diff1 value: 24.024160029526794 - type: nauc_recall_at_3_max value: 38.07599046764463 - type: nauc_recall_at_3_std value: 22.648443171847685 - type: nauc_recall_at_5_diff1 value: 21.588763686113676 - type: nauc_recall_at_5_max value: 37.16237158404055 - type: nauc_recall_at_5_std value: 24.45061830715902 - type: ndcg_at_1 value: 41.693999999999996 - type: ndcg_at_10 value: 42.047000000000004 - type: ndcg_at_100 value: 49.309 - type: ndcg_at_1000 value: 51.861999999999995 - type: ndcg_at_20 value: 44.982 - type: ndcg_at_3 value: 35.510000000000005 - type: ndcg_at_5 value: 37.529 - type: precision_at_1 value: 41.693999999999996 - type: precision_at_10 value: 13.114 - type: precision_at_100 value: 2.1069999999999998 - type: precision_at_1000 value: 0.259 - type: precision_at_20 value: 7.824000000000001 - type: precision_at_3 value: 26.796999999999997 - type: precision_at_5 value: 20.169 - type: recall_at_1 value: 18.269 - type: recall_at_10 value: 48.44 - type: recall_at_100 value: 72.909 - type: recall_at_1000 value: 86.79400000000001 - type: recall_at_20 value: 56.714 - type: recall_at_3 value: 31.85 - type: recall_at_5 value: 38.488 task: type: Retrieval - dataset: config: default name: MTEB DBPedia revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 split: test type: mteb/dbpedia metrics: - type: main_score value: 51.202000000000005 - type: map_at_1 value: 10.789 - type: map_at_10 value: 24.804000000000002 - type: map_at_100 value: 35.908 - type: map_at_1000 value: 37.97 - type: map_at_20 value: 29.236 - type: map_at_3 value: 17.055 - type: map_at_5 value: 20.342 - type: mrr_at_1 value: 78.0 - type: mrr_at_10 value: 83.82251984126987 - type: mrr_at_100 value: 84.00706659508124 - type: mrr_at_1000 value: 84.01172077534015 - type: mrr_at_20 value: 83.93946561479457 - type: mrr_at_3 value: 82.58333333333334 - type: mrr_at_5 value: 83.38333333333335 - type: nauc_map_at_1000_diff1 value: 21.975683577384412 - type: nauc_map_at_1000_max value: 33.104767603973286 - type: nauc_map_at_1000_std value: 19.507936661697688 - type: nauc_map_at_100_diff1 value: 23.19428288878281 - type: nauc_map_at_100_max value: 32.47043490749479 - type: nauc_map_at_100_std value: 16.611980248500473 - type: nauc_map_at_10_diff1 value: 23.314074413061576 - type: nauc_map_at_10_max value: 18.52506648908812 - type: nauc_map_at_10_std value: -9.718219448424597 - type: nauc_map_at_1_diff1 value: 27.402329635171146 - type: nauc_map_at_1_max value: 5.898746976402726 - type: nauc_map_at_1_std value: -26.703327281110212 - type: nauc_map_at_20_diff1 value: 23.613670514044472 - type: nauc_map_at_20_max value: 25.008187375084763 - type: nauc_map_at_20_std value: -0.05206367166066498 - type: nauc_map_at_3_diff1 value: 25.673374223753598 - type: nauc_map_at_3_max value: 12.527419567406877 - type: nauc_map_at_3_std value: -20.06963757181341 - type: nauc_map_at_5_diff1 value: 24.74002578400672 - type: nauc_map_at_5_max value: 14.23437788867648 - type: nauc_map_at_5_std value: -16.317803876665256 - type: nauc_mrr_at_1000_diff1 value: 53.26868100398232 - type: nauc_mrr_at_1000_max value: 67.65740877772801 - type: nauc_mrr_at_1000_std value: 39.43464159369656 - type: nauc_mrr_at_100_diff1 value: 53.25615896192901 - type: nauc_mrr_at_100_max value: 67.64777514366169 - type: nauc_mrr_at_100_std value: 39.410662043086006 - type: nauc_mrr_at_10_diff1 value: 52.94295111677663 - type: nauc_mrr_at_10_max value: 67.5393005296393 - type: nauc_mrr_at_10_std value: 39.31715440936177 - type: nauc_mrr_at_1_diff1 value: 57.148073445541826 - type: nauc_mrr_at_1_max value: 65.78742986970832 - type: nauc_mrr_at_1_std value: 34.198659989799246 - type: nauc_mrr_at_20_diff1 value: 53.223501273361265 - type: nauc_mrr_at_20_max value: 67.59762197314753 - type: nauc_mrr_at_20_std value: 39.359614957729356 - type: nauc_mrr_at_3_diff1 value: 53.619283112717184 - type: nauc_mrr_at_3_max value: 68.72067268448458 - type: nauc_mrr_at_3_std value: 40.53052904925793 - type: nauc_mrr_at_5_diff1 value: 52.86133436375577 - type: nauc_mrr_at_5_max value: 67.94415973414303 - type: nauc_mrr_at_5_std value: 40.09087346298919 - type: nauc_ndcg_at_1000_diff1 value: 31.008961737330505 - type: nauc_ndcg_at_1000_max value: 49.39127418414386 - type: nauc_ndcg_at_1000_std value: 37.509639671229806 - type: nauc_ndcg_at_100_diff1 value: 32.50484024525448 - type: nauc_ndcg_at_100_max value: 46.300662423725605 - type: nauc_ndcg_at_100_std value: 28.488771981297162 - type: nauc_ndcg_at_10_diff1 value: 27.911614286994414 - type: nauc_ndcg_at_10_max value: 44.70909339082426 - type: nauc_ndcg_at_10_std value: 25.644980583529154 - type: nauc_ndcg_at_1_diff1 value: 51.27342509891256 - type: nauc_ndcg_at_1_max value: 54.75803307782269 - type: nauc_ndcg_at_1_std value: 27.4853058050954 - type: nauc_ndcg_at_20_diff1 value: 30.29885920192407 - type: nauc_ndcg_at_20_max value: 43.45207612356715 - type: nauc_ndcg_at_20_std value: 21.59751863312113 - type: nauc_ndcg_at_3_diff1 value: 31.251071625533843 - type: nauc_ndcg_at_3_max value: 48.45180697571009 - type: nauc_ndcg_at_3_std value: 32.70662167853583 - type: nauc_ndcg_at_5_diff1 value: 26.175090671223877 - type: nauc_ndcg_at_5_max value: 45.2723355712432 - type: nauc_ndcg_at_5_std value: 31.461916393793 - type: nauc_precision_at_1000_diff1 value: -23.926082132378777 - type: nauc_precision_at_1000_max value: -9.350346667573811 - type: nauc_precision_at_1000_std value: 11.578726421051043 - type: nauc_precision_at_100_diff1 value: -7.468660956171794 - type: nauc_precision_at_100_max value: 19.470414434634723 - type: nauc_precision_at_100_std value: 43.86244545951367 - type: nauc_precision_at_10_diff1 value: -2.090265656696684 - type: nauc_precision_at_10_max value: 30.778228684745386 - type: nauc_precision_at_10_std value: 44.882546930240984 - type: nauc_precision_at_1_diff1 value: 57.148073445541826 - type: nauc_precision_at_1_max value: 65.78742986970832 - type: nauc_precision_at_1_std value: 34.198659989799246 - type: nauc_precision_at_20_diff1 value: -3.075798118380347 - type: nauc_precision_at_20_max value: 29.52951501638172 - type: nauc_precision_at_20_std value: 47.266521222769676 - type: nauc_precision_at_3_diff1 value: 11.892419680356198 - type: nauc_precision_at_3_max value: 43.146413741651415 - type: nauc_precision_at_3_std value: 45.2312022756118 - type: nauc_precision_at_5_diff1 value: 0.5765950918056327 - type: nauc_precision_at_5_max value: 34.22132902314228 - type: nauc_precision_at_5_std value: 44.78272426908718 - type: nauc_recall_at_1000_diff1 value: 24.99872069707702 - type: nauc_recall_at_1000_max value: 42.17319464089324 - type: nauc_recall_at_1000_std value: 47.42376725148043 - type: nauc_recall_at_100_diff1 value: 24.62929408109356 - type: nauc_recall_at_100_max value: 32.373805304406844 - type: nauc_recall_at_100_std value: 21.48682342485071 - type: nauc_recall_at_10_diff1 value: 20.62337020665992 - type: nauc_recall_at_10_max value: 14.125308316827395 - type: nauc_recall_at_10_std value: -13.565096294162865 - type: nauc_recall_at_1_diff1 value: 27.402329635171146 - type: nauc_recall_at_1_max value: 5.898746976402726 - type: nauc_recall_at_1_std value: -26.703327281110212 - type: nauc_recall_at_20_diff1 value: 22.169766882731277 - type: nauc_recall_at_20_max value: 20.588762488556828 - type: nauc_recall_at_20_std value: -4.530608772737279 - type: nauc_recall_at_3_diff1 value: 22.48622374174161 - type: nauc_recall_at_3_max value: 10.470407080375304 - type: nauc_recall_at_3_std value: -20.777479868757286 - type: nauc_recall_at_5_diff1 value: 21.28438252298866 - type: nauc_recall_at_5_max value: 10.424120660451583 - type: nauc_recall_at_5_std value: -17.912853638432384 - type: ndcg_at_1 value: 65.375 - type: ndcg_at_10 value: 51.202000000000005 - type: ndcg_at_100 value: 56.12200000000001 - type: ndcg_at_1000 value: 63.306 - type: ndcg_at_20 value: 50.442 - type: ndcg_at_3 value: 56.437000000000005 - type: ndcg_at_5 value: 53.861000000000004 - type: precision_at_1 value: 78.0 - type: precision_at_10 value: 41.075 - type: precision_at_100 value: 13.032 - type: precision_at_1000 value: 2.516 - type: precision_at_20 value: 31.4 - type: precision_at_3 value: 59.833000000000006 - type: precision_at_5 value: 51.9 - type: recall_at_1 value: 10.789 - type: recall_at_10 value: 30.059 - type: recall_at_100 value: 61.817 - type: recall_at_1000 value: 84.672 - type: recall_at_20 value: 39.135 - type: recall_at_3 value: 18.017 - type: recall_at_5 value: 22.492 task: type: Retrieval - dataset: config: default name: MTEB EmotionClassification revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 split: test type: mteb/emotion metrics: - type: accuracy value: 57.715 - type: f1 value: 51.85468544437296 - type: f1_weighted value: 58.73946069844862 - type: main_score value: 57.715 task: type: Classification - dataset: config: default name: MTEB FEVER revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 split: test type: mteb/fever metrics: - type: main_score value: 92.438 - type: map_at_1 value: 82.678 - type: map_at_10 value: 89.90899999999999 - type: map_at_100 value: 90.09899999999999 - type: map_at_1000 value: 90.11 - type: map_at_20 value: 90.026 - type: map_at_3 value: 89.034 - type: map_at_5 value: 89.619 - type: mrr_at_1 value: 89.04890489048904 - type: mrr_at_10 value: 93.53417484605598 - type: mrr_at_100 value: 93.56969053658798 - type: mrr_at_1000 value: 93.56979354808294 - type: mrr_at_20 value: 93.56100677804474 - type: mrr_at_3 value: 93.25682568256818 - type: mrr_at_5 value: 93.46909690969086 - type: nauc_map_at_1000_diff1 value: 50.19087206783256 - type: nauc_map_at_1000_max value: 26.223996443425424 - type: nauc_map_at_1000_std value: -8.531486546405336 - type: nauc_map_at_100_diff1 value: 50.12601833237827 - type: nauc_map_at_100_max value: 26.205753684531942 - type: nauc_map_at_100_std value: -8.502300882475792 - type: nauc_map_at_10_diff1 value: 49.48962186883297 - type: nauc_map_at_10_max value: 25.849578028607546 - type: nauc_map_at_10_std value: -8.58622126027856 - type: nauc_map_at_1_diff1 value: 56.88016472114475 - type: nauc_map_at_1_max value: 24.671479435457048 - type: nauc_map_at_1_std value: -11.980878470985619 - type: nauc_map_at_20_diff1 value: 49.813384246326905 - type: nauc_map_at_20_max value: 25.96508257517373 - type: nauc_map_at_20_std value: -8.568670117647939 - type: nauc_map_at_3_diff1 value: 49.087764097890165 - type: nauc_map_at_3_max value: 25.65938258554376 - type: nauc_map_at_3_std value: -8.859093431924775 - type: nauc_map_at_5_diff1 value: 49.08166208415013 - type: nauc_map_at_5_max value: 25.696246071825684 - type: nauc_map_at_5_std value: -8.431713254517472 - type: nauc_mrr_at_1000_diff1 value: 73.35484368612293 - type: nauc_mrr_at_1000_max value: 35.657386688053336 - type: nauc_mrr_at_1000_std value: -18.09172713569766 - type: nauc_mrr_at_100_diff1 value: 73.35508125874483 - type: nauc_mrr_at_100_max value: 35.65842743437027 - type: nauc_mrr_at_100_std value: -18.08981699366641 - type: nauc_mrr_at_10_diff1 value: 73.29004337552368 - type: nauc_mrr_at_10_max value: 35.882001444609216 - type: nauc_mrr_at_10_std value: -18.05339396879553 - type: nauc_mrr_at_1_diff1 value: 74.48742882702338 - type: nauc_mrr_at_1_max value: 31.49138530538466 - type: nauc_mrr_at_1_std value: -19.510294856397955 - type: nauc_mrr_at_20_diff1 value: 73.3388656330962 - type: nauc_mrr_at_20_max value: 35.706948273788505 - type: nauc_mrr_at_20_std value: -18.140154123750992 - type: nauc_mrr_at_3_diff1 value: 73.22698350499 - type: nauc_mrr_at_3_max value: 36.4855373316516 - type: nauc_mrr_at_3_std value: -17.719256990311198 - type: nauc_mrr_at_5_diff1 value: 73.24460108538948 - type: nauc_mrr_at_5_max value: 36.322370705490634 - type: nauc_mrr_at_5_std value: -17.636279233457984 - type: nauc_ndcg_at_1000_diff1 value: 53.674109881592756 - type: nauc_ndcg_at_1000_max value: 28.767387846727487 - type: nauc_ndcg_at_1000_std value: -8.858681782014946 - type: nauc_ndcg_at_100_diff1 value: 52.33608078847966 - type: nauc_ndcg_at_100_max value: 28.511414384159877 - type: nauc_ndcg_at_100_std value: -8.085385430073922 - type: nauc_ndcg_at_10_diff1 value: 49.712295545440774 - type: nauc_ndcg_at_10_max value: 27.5674225152019 - type: nauc_ndcg_at_10_std value: -8.244677630275376 - type: nauc_ndcg_at_1_diff1 value: 74.48742882702338 - type: nauc_ndcg_at_1_max value: 31.49138530538466 - type: nauc_ndcg_at_1_std value: -19.510294856397955 - type: nauc_ndcg_at_20_diff1 value: 50.61628846813059 - type: nauc_ndcg_at_20_max value: 27.53989784238201 - type: nauc_ndcg_at_20_std value: -8.373695482986479 - type: nauc_ndcg_at_3_diff1 value: 51.295860863016884 - type: nauc_ndcg_at_3_max value: 28.99776689198307 - type: nauc_ndcg_at_3_std value: -8.878181909861983 - type: nauc_ndcg_at_5_diff1 value: 49.619081645734504 - type: nauc_ndcg_at_5_max value: 28.11109235395876 - type: nauc_ndcg_at_5_std value: -7.722157727171728 - type: nauc_precision_at_1000_diff1 value: -9.298540465937485 - type: nauc_precision_at_1000_max value: -1.3157308795912563 - type: nauc_precision_at_1000_std value: 1.897355386264135 - type: nauc_precision_at_100_diff1 value: -12.246672190804334 - type: nauc_precision_at_100_max value: -0.9687067276412682 - type: nauc_precision_at_100_std value: 4.56074518564851 - type: nauc_precision_at_10_diff1 value: -15.533411370200923 - type: nauc_precision_at_10_max value: -2.191843047666222 - type: nauc_precision_at_10_std value: 3.6723841478730748 - type: nauc_precision_at_1_diff1 value: 74.48742882702338 - type: nauc_precision_at_1_max value: 31.49138530538466 - type: nauc_precision_at_1_std value: -19.510294856397955 - type: nauc_precision_at_20_diff1 value: -15.290364061922347 - type: nauc_precision_at_20_max value: -2.921722171191804 - type: nauc_precision_at_20_std value: 4.08482465973661 - type: nauc_precision_at_3_diff1 value: -8.208906597107383 - type: nauc_precision_at_3_max value: 2.9796478961627284 - type: nauc_precision_at_3_std value: 0.34366033602604895 - type: nauc_precision_at_5_diff1 value: -14.42241522747573 - type: nauc_precision_at_5_max value: -0.5633890785935999 - type: nauc_precision_at_5_std value: 3.7064496791809836 - type: nauc_recall_at_1000_diff1 value: -0.5673198466803553 - type: nauc_recall_at_1000_max value: 21.92110385096128 - type: nauc_recall_at_1000_std value: 54.421987386115475 - type: nauc_recall_at_100_diff1 value: -0.6512704079314391 - type: nauc_recall_at_100_max value: 22.38252665262688 - type: nauc_recall_at_100_std value: 36.50750378730013 - type: nauc_recall_at_10_diff1 value: 11.308848658347774 - type: nauc_recall_at_10_max value: 21.077700181656738 - type: nauc_recall_at_10_std value: 8.321338697504787 - type: nauc_recall_at_1_diff1 value: 56.88016472114475 - type: nauc_recall_at_1_max value: 24.671479435457048 - type: nauc_recall_at_1_std value: -11.980878470985619 - type: nauc_recall_at_20_diff1 value: 5.8415071379210755 - type: nauc_recall_at_20_max value: 16.97886837481554 - type: nauc_recall_at_20_std value: 12.529145693495494 - type: nauc_recall_at_3_diff1 value: 27.396913234035086 - type: nauc_recall_at_3_max value: 24.897648442357994 - type: nauc_recall_at_3_std value: 0.8528297027573939 - type: nauc_recall_at_5_diff1 value: 18.295838017557397 - type: nauc_recall_at_5_max value: 24.077879268127823 - type: nauc_recall_at_5_std value: 7.099403908855888 - type: ndcg_at_1 value: 89.049 - type: ndcg_at_10 value: 92.438 - type: ndcg_at_100 value: 93.016 - type: ndcg_at_1000 value: 93.17699999999999 - type: ndcg_at_20 value: 92.713 - type: ndcg_at_3 value: 91.40599999999999 - type: ndcg_at_5 value: 92.026 - type: precision_at_1 value: 89.049 - type: precision_at_10 value: 10.917 - type: precision_at_100 value: 1.146 - type: precision_at_1000 value: 0.117 - type: precision_at_20 value: 5.56 - type: precision_at_3 value: 34.598 - type: precision_at_5 value: 21.323 - type: recall_at_1 value: 82.678 - type: recall_at_10 value: 96.465 - type: recall_at_100 value: 98.571 - type: recall_at_1000 value: 99.496 - type: recall_at_20 value: 97.342 - type: recall_at_3 value: 93.696 - type: recall_at_5 value: 95.324 task: type: Retrieval - dataset: config: default name: MTEB FiQA2018 revision: 27a168819829fe9bcd655c2df245fb19452e8e06 split: test type: mteb/fiqa metrics: - type: main_score value: 58.781000000000006 - type: map_at_1 value: 31.107000000000003 - type: map_at_10 value: 50.955 - type: map_at_100 value: 53.177 - type: map_at_1000 value: 53.291 - type: map_at_20 value: 52.271 - type: map_at_3 value: 44.762 - type: map_at_5 value: 48.379 - type: mrr_at_1 value: 58.48765432098766 - type: mrr_at_10 value: 66.11429551244368 - type: mrr_at_100 value: 66.68929754431386 - type: mrr_at_1000 value: 66.71304995006113 - type: mrr_at_20 value: 66.46821550477237 - type: mrr_at_3 value: 64.22325102880657 - type: mrr_at_5 value: 65.41152263374484 - type: nauc_map_at_1000_diff1 value: 45.490146083064445 - type: nauc_map_at_1000_max value: 33.573139354617126 - type: nauc_map_at_1000_std value: -14.07140489937541 - type: nauc_map_at_100_diff1 value: 45.48828357408913 - type: nauc_map_at_100_max value: 33.51907944260763 - type: nauc_map_at_100_std value: -14.059609883903152 - type: nauc_map_at_10_diff1 value: 45.70748844526757 - type: nauc_map_at_10_max value: 31.667587503334587 - type: nauc_map_at_10_std value: -15.076948336390855 - type: nauc_map_at_1_diff1 value: 51.42775649850064 - type: nauc_map_at_1_max value: 16.56862308325116 - type: nauc_map_at_1_std value: -14.684731980257675 - type: nauc_map_at_20_diff1 value: 45.754998522906284 - type: nauc_map_at_20_max value: 33.03759060247343 - type: nauc_map_at_20_std value: -14.750787459968736 - type: nauc_map_at_3_diff1 value: 46.45241223088609 - type: nauc_map_at_3_max value: 26.607789112226467 - type: nauc_map_at_3_std value: -14.997049792585598 - type: nauc_map_at_5_diff1 value: 45.87702900983919 - type: nauc_map_at_5_max value: 30.076255479914348 - type: nauc_map_at_5_std value: -15.062787509367553 - type: nauc_mrr_at_1000_diff1 value: 55.64889336097758 - type: nauc_mrr_at_1000_max value: 48.57022261913911 - type: nauc_mrr_at_1000_std value: -12.428435474800143 - type: nauc_mrr_at_100_diff1 value: 55.62957328562593 - type: nauc_mrr_at_100_max value: 48.56575267775789 - type: nauc_mrr_at_100_std value: -12.415226616847987 - type: nauc_mrr_at_10_diff1 value: 55.5931002027865 - type: nauc_mrr_at_10_max value: 48.428200063552374 - type: nauc_mrr_at_10_std value: -12.590361961152267 - type: nauc_mrr_at_1_diff1 value: 59.470635489729105 - type: nauc_mrr_at_1_max value: 49.66866699872627 - type: nauc_mrr_at_1_std value: -13.590112604913607 - type: nauc_mrr_at_20_diff1 value: 55.60145155716686 - type: nauc_mrr_at_20_max value: 48.58677663675733 - type: nauc_mrr_at_20_std value: -12.454093344399036 - type: nauc_mrr_at_3_diff1 value: 55.76657118158415 - type: nauc_mrr_at_3_max value: 48.88547787372198 - type: nauc_mrr_at_3_std value: -13.299744066289124 - type: nauc_mrr_at_5_diff1 value: 55.55217612731964 - type: nauc_mrr_at_5_max value: 48.56957852769844 - type: nauc_mrr_at_5_std value: -12.876904435466624 - type: nauc_ndcg_at_1000_diff1 value: 47.2645656074121 - type: nauc_ndcg_at_1000_max value: 39.95808937564202 - type: nauc_ndcg_at_1000_std value: -11.366829207572232 - type: nauc_ndcg_at_100_diff1 value: 46.89043419464991 - type: nauc_ndcg_at_100_max value: 39.00034359981605 - type: nauc_ndcg_at_100_std value: -10.697277437129921 - type: nauc_ndcg_at_10_diff1 value: 47.07625032910763 - type: nauc_ndcg_at_10_max value: 35.51275239983428 - type: nauc_ndcg_at_10_std value: -13.965305287946128 - type: nauc_ndcg_at_1_diff1 value: 59.470635489729105 - type: nauc_ndcg_at_1_max value: 49.66866699872627 - type: nauc_ndcg_at_1_std value: -13.590112604913607 - type: nauc_ndcg_at_20_diff1 value: 47.44262917418296 - type: nauc_ndcg_at_20_max value: 37.6804112715633 - type: nauc_ndcg_at_20_std value: -13.174880813005297 - type: nauc_ndcg_at_3_diff1 value: 44.56982475937759 - type: nauc_ndcg_at_3_max value: 37.96424549723314 - type: nauc_ndcg_at_3_std value: -13.657607148249964 - type: nauc_ndcg_at_5_diff1 value: 45.427291740214024 - type: nauc_ndcg_at_5_max value: 35.42232275517991 - type: nauc_ndcg_at_5_std value: -14.510048307634808 - type: nauc_precision_at_1000_diff1 value: -16.58479747595096 - type: nauc_precision_at_1000_max value: 27.22386867486023 - type: nauc_precision_at_1000_std value: 9.41210384044254 - type: nauc_precision_at_100_diff1 value: -11.640382840009572 - type: nauc_precision_at_100_max value: 30.20752947841474 - type: nauc_precision_at_100_std value: 10.72773947232612 - type: nauc_precision_at_10_diff1 value: 3.2540578244055594 - type: nauc_precision_at_10_max value: 35.80515547017638 - type: nauc_precision_at_10_std value: 0.299517152086918 - type: nauc_precision_at_1_diff1 value: 59.470635489729105 - type: nauc_precision_at_1_max value: 49.66866699872627 - type: nauc_precision_at_1_std value: -13.590112604913607 - type: nauc_precision_at_20_diff1 value: -1.8627219860435185 - type: nauc_precision_at_20_max value: 35.9181314633325 - type: nauc_precision_at_20_std value: 4.491869749000042 - type: nauc_precision_at_3_diff1 value: 17.94168903901189 - type: nauc_precision_at_3_max value: 41.67388438464254 - type: nauc_precision_at_3_std value: -5.38615084998387 - type: nauc_precision_at_5_diff1 value: 9.312012525324068 - type: nauc_precision_at_5_max value: 39.52463080415461 - type: nauc_precision_at_5_std value: -2.615286156278468 - type: nauc_recall_at_1000_diff1 value: 37.0960616996064 - type: nauc_recall_at_1000_max value: 46.91967503624078 - type: nauc_recall_at_1000_std value: 36.70723581015844 - type: nauc_recall_at_100_diff1 value: 32.54497560045993 - type: nauc_recall_at_100_max value: 26.846226776082734 - type: nauc_recall_at_100_std value: 9.257918182671672 - type: nauc_recall_at_10_diff1 value: 40.05619869408745 - type: nauc_recall_at_10_max value: 25.504319960057014 - type: nauc_recall_at_10_std value: -12.57012842016253 - type: nauc_recall_at_1_diff1 value: 51.42775649850064 - type: nauc_recall_at_1_max value: 16.56862308325116 - type: nauc_recall_at_1_std value: -14.684731980257675 - type: nauc_recall_at_20_diff1 value: 39.34128607816815 - type: nauc_recall_at_20_max value: 28.31147877410395 - type: nauc_recall_at_20_std value: -10.295180225906224 - type: nauc_recall_at_3_diff1 value: 41.31333745355922 - type: nauc_recall_at_3_max value: 22.642649370921276 - type: nauc_recall_at_3_std value: -14.44811859378254 - type: nauc_recall_at_5_diff1 value: 39.91795256714951 - type: nauc_recall_at_5_max value: 24.396817798634245 - type: nauc_recall_at_5_std value: -13.696077909471175 - type: ndcg_at_1 value: 58.48799999999999 - type: ndcg_at_10 value: 58.781000000000006 - type: ndcg_at_100 value: 65.212 - type: ndcg_at_1000 value: 66.85900000000001 - type: ndcg_at_20 value: 61.529999999999994 - type: ndcg_at_3 value: 54.864000000000004 - type: ndcg_at_5 value: 56.223 - type: precision_at_1 value: 58.48799999999999 - type: precision_at_10 value: 16.111 - type: precision_at_100 value: 2.298 - type: precision_at_1000 value: 0.257 - type: precision_at_20 value: 9.306000000000001 - type: precision_at_3 value: 36.317 - type: precision_at_5 value: 26.759 - type: recall_at_1 value: 31.107000000000003 - type: recall_at_10 value: 65.08500000000001 - type: recall_at_100 value: 87.91 - type: recall_at_1000 value: 97.817 - type: recall_at_20 value: 73.282 - type: recall_at_3 value: 49.317 - type: recall_at_5 value: 56.617 task: type: Retrieval - dataset: config: default name: MTEB HotpotQA revision: ab518f4d6fcca38d87c25209f94beba119d02014 split: test type: mteb/hotpotqa metrics: - type: main_score value: 79.449 - type: map_at_1 value: 41.384 - type: map_at_10 value: 72.844 - type: map_at_100 value: 73.589 - type: map_at_1000 value: 73.624 - type: map_at_20 value: 73.317 - type: map_at_3 value: 69.427 - type: map_at_5 value: 71.68299999999999 - type: mrr_at_1 value: 82.76839972991222 - type: mrr_at_10 value: 87.92045807744634 - type: mrr_at_100 value: 88.05529589670978 - type: mrr_at_1000 value: 88.05935891074716 - type: mrr_at_20 value: 88.01402112970962 - type: mrr_at_3 value: 87.3216295295969 - type: mrr_at_5 value: 87.69705154175074 - type: nauc_map_at_1000_diff1 value: 20.396629535039704 - type: nauc_map_at_1000_max value: 39.10949908339265 - type: nauc_map_at_1000_std value: 10.224729673688502 - type: nauc_map_at_100_diff1 value: 20.381077063965574 - type: nauc_map_at_100_max value: 39.12262980169527 - type: nauc_map_at_100_std value: 10.256952440972226 - type: nauc_map_at_10_diff1 value: 20.227214487416916 - type: nauc_map_at_10_max value: 39.065878364926085 - type: nauc_map_at_10_std value: 9.830819360569484 - type: nauc_map_at_1_diff1 value: 60.61929089121275 - type: nauc_map_at_1_max value: 49.53547409224507 - type: nauc_map_at_1_std value: 0.2722096857291782 - type: nauc_map_at_20_diff1 value: 20.183915365850165 - type: nauc_map_at_20_max value: 39.06905710390586 - type: nauc_map_at_20_std value: 10.244769286257812 - type: nauc_map_at_3_diff1 value: 18.953350220177363 - type: nauc_map_at_3_max value: 36.89647666189664 - type: nauc_map_at_3_std value: 6.856939205711613 - type: nauc_map_at_5_diff1 value: 19.74313508534105 - type: nauc_map_at_5_max value: 38.42860611762909 - type: nauc_map_at_5_std value: 8.620757357067802 - type: nauc_mrr_at_1000_diff1 value: 60.544760748070416 - type: nauc_mrr_at_1000_max value: 53.536700750600176 - type: nauc_mrr_at_1000_std value: 4.356103341419562 - type: nauc_mrr_at_100_diff1 value: 60.543037850402825 - type: nauc_mrr_at_100_max value: 53.54473925679791 - type: nauc_mrr_at_100_std value: 4.3713759172294475 - type: nauc_mrr_at_10_diff1 value: 60.57585979923885 - type: nauc_mrr_at_10_max value: 53.65882404973961 - type: nauc_mrr_at_10_std value: 4.46866142907982 - type: nauc_mrr_at_1_diff1 value: 60.61929089121275 - type: nauc_mrr_at_1_max value: 49.53547409224507 - type: nauc_mrr_at_1_std value: 0.2722096857291782 - type: nauc_mrr_at_20_diff1 value: 60.541893232518674 - type: nauc_mrr_at_20_max value: 53.6135776399171 - type: nauc_mrr_at_20_std value: 4.443552945861195 - type: nauc_mrr_at_3_diff1 value: 60.46996364153697 - type: nauc_mrr_at_3_max value: 53.981024588336936 - type: nauc_mrr_at_3_std value: 4.300285863686253 - type: nauc_mrr_at_5_diff1 value: 60.562791070200426 - type: nauc_mrr_at_5_max value: 53.884058343579966 - type: nauc_mrr_at_5_std value: 4.35333313705802 - type: nauc_ndcg_at_1000_diff1 value: 26.909558826785485 - type: nauc_ndcg_at_1000_max value: 43.2090252545764 - type: nauc_ndcg_at_1000_std value: 13.24632397019833 - type: nauc_ndcg_at_100_diff1 value: 26.4096138903785 - type: nauc_ndcg_at_100_max value: 43.50667894420325 - type: nauc_ndcg_at_100_std value: 14.272929786830657 - type: nauc_ndcg_at_10_diff1 value: 25.261392560708607 - type: nauc_ndcg_at_10_max value: 43.02496845139645 - type: nauc_ndcg_at_10_std value: 12.753991213996402 - type: nauc_ndcg_at_1_diff1 value: 60.61929089121275 - type: nauc_ndcg_at_1_max value: 49.53547409224507 - type: nauc_ndcg_at_1_std value: 0.2722096857291782 - type: nauc_ndcg_at_20_diff1 value: 25.15730629354081 - type: nauc_ndcg_at_20_max value: 43.10358742768409 - type: nauc_ndcg_at_20_std value: 14.103247675055986 - type: nauc_ndcg_at_3_diff1 value: 23.492158440363873 - type: nauc_ndcg_at_3_max value: 39.880317429264736 - type: nauc_ndcg_at_3_std value: 7.852278799949863 - type: nauc_ndcg_at_5_diff1 value: 24.46471897598423 - type: nauc_ndcg_at_5_max value: 41.901821932685294 - type: nauc_ndcg_at_5_std value: 10.33482164145028 - type: nauc_precision_at_1000_diff1 value: 14.556112531859444 - type: nauc_precision_at_1000_max value: 54.51236512101235 - type: nauc_precision_at_1000_std value: 68.89420216988455 - type: nauc_precision_at_100_diff1 value: 14.116319404924122 - type: nauc_precision_at_100_max value: 50.42943334977378 - type: nauc_precision_at_100_std value: 49.80016017936658 - type: nauc_precision_at_10_diff1 value: 14.530495877243805 - type: nauc_precision_at_10_max value: 43.89651175033577 - type: nauc_precision_at_10_std value: 24.764789718434958 - type: nauc_precision_at_1_diff1 value: 60.61929089121275 - type: nauc_precision_at_1_max value: 49.53547409224507 - type: nauc_precision_at_1_std value: 0.2722096857291782 - type: nauc_precision_at_20_diff1 value: 11.499635650364958 - type: nauc_precision_at_20_max value: 44.499499741252265 - type: nauc_precision_at_20_std value: 33.743842605352725 - type: nauc_precision_at_3_diff1 value: 14.621019803797811 - type: nauc_precision_at_3_max value: 38.1391146398071 - type: nauc_precision_at_3_std value: 11.050680597126348 - type: nauc_precision_at_5_diff1 value: 14.878056511475538 - type: nauc_precision_at_5_max value: 41.52854585813069 - type: nauc_precision_at_5_std value: 16.596884488946877 - type: nauc_recall_at_1000_diff1 value: 14.556112531860405 - type: nauc_recall_at_1000_max value: 54.512365121012444 - type: nauc_recall_at_1000_std value: 68.89420216988472 - type: nauc_recall_at_100_diff1 value: 14.11631940492389 - type: nauc_recall_at_100_max value: 50.42943334977325 - type: nauc_recall_at_100_std value: 49.80016017936635 - type: nauc_recall_at_10_diff1 value: 14.530495877243975 - type: nauc_recall_at_10_max value: 43.89651175033581 - type: nauc_recall_at_10_std value: 24.764789718434855 - type: nauc_recall_at_1_diff1 value: 60.61929089121275 - type: nauc_recall_at_1_max value: 49.53547409224507 - type: nauc_recall_at_1_std value: 0.2722096857291782 - type: nauc_recall_at_20_diff1 value: 11.499635650364953 - type: nauc_recall_at_20_max value: 44.499499741252166 - type: nauc_recall_at_20_std value: 33.74384260535269 - type: nauc_recall_at_3_diff1 value: 14.621019803797758 - type: nauc_recall_at_3_max value: 38.139114639807104 - type: nauc_recall_at_3_std value: 11.050680597126208 - type: nauc_recall_at_5_diff1 value: 14.87805651147543 - type: nauc_recall_at_5_max value: 41.52854585813069 - type: nauc_recall_at_5_std value: 16.59688448894684 - type: ndcg_at_1 value: 82.768 - type: ndcg_at_10 value: 79.449 - type: ndcg_at_100 value: 81.878 - type: ndcg_at_1000 value: 82.526 - type: ndcg_at_20 value: 80.601 - type: ndcg_at_3 value: 74.899 - type: ndcg_at_5 value: 77.586 - type: precision_at_1 value: 82.768 - type: precision_at_10 value: 16.804 - type: precision_at_100 value: 1.8659999999999999 - type: precision_at_1000 value: 0.19499999999999998 - type: precision_at_20 value: 8.770999999999999 - type: precision_at_3 value: 49.381 - type: precision_at_5 value: 31.746000000000002 - type: recall_at_1 value: 41.384 - type: recall_at_10 value: 84.018 - type: recall_at_100 value: 93.30199999999999 - type: recall_at_1000 value: 97.529 - type: recall_at_20 value: 87.711 - type: recall_at_3 value: 74.072 - type: recall_at_5 value: 79.365 task: type: Retrieval - dataset: config: default name: MTEB ImdbClassification revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 split: test type: mteb/imdb metrics: - type: accuracy value: 95.50119999999997 - type: ap value: 93.27855740989341 - type: ap_weighted value: 93.27855740989341 - type: f1 value: 95.49922732391366 - type: f1_weighted value: 95.49922732391366 - type: main_score value: 95.50119999999997 task: type: Classification - dataset: config: default name: MTEB MSMARCO revision: c5a29a104738b98a9e76336939199e264163d4a0 split: dev type: mteb/msmarco metrics: - type: main_score value: 44.181 - type: map_at_1 value: 24.3 - type: map_at_10 value: 37.064 - type: map_at_100 value: 38.217 - type: map_at_1000 value: 38.261 - type: map_at_20 value: 37.797 - type: map_at_3 value: 33.03 - type: map_at_5 value: 35.382000000000005 - type: mrr_at_1 value: 25.014326647564474 - type: mrr_at_10 value: 37.67002092145362 - type: mrr_at_100 value: 38.76618716955713 - type: mrr_at_1000 value: 38.803895343578624 - type: mrr_at_20 value: 38.372875531879025 - type: mrr_at_3 value: 33.74164278892073 - type: mrr_at_5 value: 36.04250238777461 - type: nauc_map_at_1000_diff1 value: 37.38914109165067 - type: nauc_map_at_1000_max value: 9.290439022090213 - type: nauc_map_at_1000_std value: -17.68507604596775 - type: nauc_map_at_100_diff1 value: 37.3858106261435 - type: nauc_map_at_100_max value: 9.292194370842791 - type: nauc_map_at_100_std value: -17.6461510679294 - type: nauc_map_at_10_diff1 value: 37.24355836056403 - type: nauc_map_at_10_max value: 9.19029394636661 - type: nauc_map_at_10_std value: -18.216369315567626 - type: nauc_map_at_1_diff1 value: 40.298938486026984 - type: nauc_map_at_1_max value: 8.149499405622326 - type: nauc_map_at_1_std value: -17.09168853307602 - type: nauc_map_at_20_diff1 value: 37.344123641575216 - type: nauc_map_at_20_max value: 9.24559383901809 - type: nauc_map_at_20_std value: -17.842740773642962 - type: nauc_map_at_3_diff1 value: 37.4023127968177 - type: nauc_map_at_3_max value: 8.930674077317596 - type: nauc_map_at_3_std value: -18.68520909934096 - type: nauc_map_at_5_diff1 value: 37.12600186091895 - type: nauc_map_at_5_max value: 9.173506919924861 - type: nauc_map_at_5_std value: -18.625677130615294 - type: nauc_mrr_at_1000_diff1 value: 37.34256456294692 - type: nauc_mrr_at_1000_max value: 9.276741130450404 - type: nauc_mrr_at_1000_std value: -17.41693013754444 - type: nauc_mrr_at_100_diff1 value: 37.33775949993714 - type: nauc_mrr_at_100_max value: 9.28051163202218 - type: nauc_mrr_at_100_std value: -17.381741706111445 - type: nauc_mrr_at_10_diff1 value: 37.21506505847139 - type: nauc_mrr_at_10_max value: 9.200324529184542 - type: nauc_mrr_at_10_std value: -17.904321523440817 - type: nauc_mrr_at_1_diff1 value: 40.314678345050915 - type: nauc_mrr_at_1_max value: 8.193685362111243 - type: nauc_mrr_at_1_std value: -17.096535887474175 - type: nauc_mrr_at_20_diff1 value: 37.293746882874004 - type: nauc_mrr_at_20_max value: 9.256273923676206 - type: nauc_mrr_at_20_std value: -17.528338232043577 - type: nauc_mrr_at_3_diff1 value: 37.254812254578376 - type: nauc_mrr_at_3_max value: 8.903676300128614 - type: nauc_mrr_at_3_std value: -18.49940979312031 - type: nauc_mrr_at_5_diff1 value: 37.08969825523026 - type: nauc_mrr_at_5_max value: 9.194982897416688 - type: nauc_mrr_at_5_std value: -18.291840579141315 - type: nauc_ndcg_at_1000_diff1 value: 36.930810397557096 - type: nauc_ndcg_at_1000_max value: 9.8356345032183 - type: nauc_ndcg_at_1000_std value: -16.308145152943887 - type: nauc_ndcg_at_100_diff1 value: 36.901149744427414 - type: nauc_ndcg_at_100_max value: 9.96065454342114 - type: nauc_ndcg_at_100_std value: -14.983815239399584 - type: nauc_ndcg_at_10_diff1 value: 36.441571794416724 - type: nauc_ndcg_at_10_max value: 9.57337658776914 - type: nauc_ndcg_at_10_std value: -17.88037638294921 - type: nauc_ndcg_at_1_diff1 value: 40.314678345050915 - type: nauc_ndcg_at_1_max value: 8.193685362111243 - type: nauc_ndcg_at_1_std value: -17.096535887474175 - type: nauc_ndcg_at_20_diff1 value: 36.775334219857484 - type: nauc_ndcg_at_20_max value: 9.789544462660507 - type: nauc_ndcg_at_20_std value: -16.465733594062474 - type: nauc_ndcg_at_3_diff1 value: 36.58838956901628 - type: nauc_ndcg_at_3_max value: 9.089768089567865 - type: nauc_ndcg_at_3_std value: -19.12823913473232 - type: nauc_ndcg_at_5_diff1 value: 36.147729725463364 - type: nauc_ndcg_at_5_max value: 9.53707003144017 - type: nauc_ndcg_at_5_std value: -18.91372487441106 - type: nauc_precision_at_1000_diff1 value: -6.013504255890098 - type: nauc_precision_at_1000_max value: 6.319348588937731 - type: nauc_precision_at_1000_std value: 6.360339202992953 - type: nauc_precision_at_100_diff1 value: 14.846649240680357 - type: nauc_precision_at_100_max value: 11.751644343520605 - type: nauc_precision_at_100_std value: 16.881205928162444 - type: nauc_precision_at_10_diff1 value: 30.328513184776966 - type: nauc_precision_at_10_max value: 9.988509735977631 - type: nauc_precision_at_10_std value: -15.609966599969837 - type: nauc_precision_at_1_diff1 value: 40.314678345050915 - type: nauc_precision_at_1_max value: 8.193685362111243 - type: nauc_precision_at_1_std value: -17.096535887474175 - type: nauc_precision_at_20_diff1 value: 28.248245250811543 - type: nauc_precision_at_20_max value: 10.953279209883918 - type: nauc_precision_at_20_std value: -7.365540710727016 - type: nauc_precision_at_3_diff1 value: 33.6150964111514 - type: nauc_precision_at_3_max value: 9.216455510763346 - type: nauc_precision_at_3_std value: -20.45932513010908 - type: nauc_precision_at_5_diff1 value: 31.518755311864705 - type: nauc_precision_at_5_max value: 10.019710006442747 - type: nauc_precision_at_5_std value: -19.740528698385468 - type: nauc_recall_at_1000_diff1 value: 12.207155589507542 - type: nauc_recall_at_1000_max value: 39.3447783153665 - type: nauc_recall_at_1000_std value: 74.60352827999826 - type: nauc_recall_at_100_diff1 value: 32.993666280768615 - type: nauc_recall_at_100_max value: 16.487188889720816 - type: nauc_recall_at_100_std value: 26.828206265371275 - type: nauc_recall_at_10_diff1 value: 33.65453771237772 - type: nauc_recall_at_10_max value: 10.71869814574723 - type: nauc_recall_at_10_std value: -16.27859785753318 - type: nauc_recall_at_1_diff1 value: 40.298938486026984 - type: nauc_recall_at_1_max value: 8.149499405622326 - type: nauc_recall_at_1_std value: -17.09168853307602 - type: nauc_recall_at_20_diff1 value: 34.60034971417269 - type: nauc_recall_at_20_max value: 12.076871992384788 - type: nauc_recall_at_20_std value: -8.224571589978806 - type: nauc_recall_at_3_diff1 value: 34.24661417034744 - type: nauc_recall_at_3_max value: 9.464103325281997 - type: nauc_recall_at_3_std value: -20.329748455626195 - type: nauc_recall_at_5_diff1 value: 33.042225241281585 - type: nauc_recall_at_5_max value: 10.486814885646142 - type: nauc_recall_at_5_std value: -19.7259662900716 - type: ndcg_at_1 value: 25.013999999999996 - type: ndcg_at_10 value: 44.181 - type: ndcg_at_100 value: 49.673 - type: ndcg_at_1000 value: 50.705999999999996 - type: ndcg_at_20 value: 46.798 - type: ndcg_at_3 value: 36.037 - type: ndcg_at_5 value: 40.214 - type: precision_at_1 value: 25.013999999999996 - type: precision_at_10 value: 6.9110000000000005 - type: precision_at_100 value: 0.9650000000000001 - type: precision_at_1000 value: 0.105 - type: precision_at_20 value: 4.004 - type: precision_at_3 value: 15.238999999999999 - type: precision_at_5 value: 11.264000000000001 - type: recall_at_1 value: 24.3 - type: recall_at_10 value: 66.06400000000001 - type: recall_at_100 value: 91.291 - type: recall_at_1000 value: 99.054 - type: recall_at_20 value: 76.25699999999999 - type: recall_at_3 value: 44.039 - type: recall_at_5 value: 54.053 task: type: Retrieval - dataset: config: en name: MTEB MTOPDomainClassification (en) revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf split: test type: mteb/mtop_domain metrics: - type: accuracy value: 96.35430916552667 - type: f1 value: 96.16669219074517 - type: f1_weighted value: 96.35506582065435 - type: main_score value: 96.35430916552667 task: type: Classification - dataset: config: en name: MTEB MTOPIntentClassification (en) revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba split: test type: mteb/mtop_intent metrics: - type: accuracy value: 80.51527587779297 - type: f1 value: 59.350461259612345 - type: f1_weighted value: 81.51891267687044 - type: main_score value: 80.51527587779297 task: type: Classification - dataset: config: en name: MTEB MassiveIntentClassification (en) revision: 4672e20407010da34463acc759c162ca9734bca6 split: test type: mteb/amazon_massive_intent metrics: - type: accuracy value: 77.31338264963013 - type: f1 value: 75.29547524788576 - type: f1_weighted value: 76.26831259224058 - type: main_score value: 77.31338264963013 task: type: Classification - dataset: config: en name: MTEB MassiveScenarioClassification (en) revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 split: test type: mteb/amazon_massive_scenario metrics: - type: accuracy value: 79.97982515131137 - type: f1 value: 79.34057805450769 - type: f1_weighted value: 79.73023446597212 - type: main_score value: 79.97982515131137 task: type: Classification - dataset: config: default name: MTEB MedrxivClusteringP2P revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 split: test type: mteb/medrxiv-clustering-p2p metrics: - type: main_score value: 38.37635785818304 - type: v_measure value: 38.37635785818304 - type: v_measure_std value: 1.6943794496059137 task: type: Clustering - dataset: config: default name: MTEB MedrxivClusteringS2S revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 split: test type: mteb/medrxiv-clustering-s2s metrics: - type: main_score value: 37.6711034083755 - type: v_measure value: 37.6711034083755 - type: v_measure_std value: 1.1408887612104992 task: type: Clustering - dataset: config: default name: MTEB MindSmallReranking revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 split: test type: mteb/mind_small metrics: - type: main_score value: 32.20170969306457 - type: map value: 32.20170969306457 - type: mrr value: 33.41738896071552 - type: nAUC_map_diff1 value: 12.077124363492512 - type: nAUC_map_max value: -20.336429990396454 - type: nAUC_map_std value: 0.10724031251638018 - type: nAUC_mrr_diff1 value: 11.405695518900744 - type: nAUC_mrr_max value: -15.0727490448132 - type: nAUC_mrr_std value: 1.8987958512727106 task: type: Reranking - dataset: config: default name: MTEB NFCorpus revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 split: test type: mteb/nfcorpus metrics: - type: main_score value: 40.463 - type: map_at_1 value: 6.4990000000000006 - type: map_at_10 value: 15.699 - type: map_at_100 value: 19.895 - type: map_at_1000 value: 21.537 - type: map_at_20 value: 17.429 - type: map_at_3 value: 11.48 - type: map_at_5 value: 13.383999999999999 - type: mrr_at_1 value: 52.63157894736842 - type: mrr_at_10 value: 61.60401002506265 - type: mrr_at_100 value: 62.04336653809741 - type: mrr_at_1000 value: 62.07610833363911 - type: mrr_at_20 value: 61.88033067968176 - type: mrr_at_3 value: 59.44272445820435 - type: mrr_at_5 value: 60.89783281733746 - type: nauc_map_at_1000_diff1 value: 18.58585974547791 - type: nauc_map_at_1000_max value: 30.25465935470905 - type: nauc_map_at_1000_std value: 10.987080017051682 - type: nauc_map_at_100_diff1 value: 20.02651798573329 - type: nauc_map_at_100_max value: 30.108719787095467 - type: nauc_map_at_100_std value: 7.882019722247158 - type: nauc_map_at_10_diff1 value: 23.02800157136177 - type: nauc_map_at_10_max value: 22.8723397741279 - type: nauc_map_at_10_std value: -3.762893117006399 - type: nauc_map_at_1_diff1 value: 37.94611136294878 - type: nauc_map_at_1_max value: 7.297492349938244 - type: nauc_map_at_1_std value: -17.813930346562152 - type: nauc_map_at_20_diff1 value: 21.981440837881113 - type: nauc_map_at_20_max value: 26.759497880383837 - type: nauc_map_at_20_std value: 0.18040330674839283 - type: nauc_map_at_3_diff1 value: 27.066009968256555 - type: nauc_map_at_3_max value: 10.488797596450187 - type: nauc_map_at_3_std value: -14.013059830876845 - type: nauc_map_at_5_diff1 value: 25.493785001708446 - type: nauc_map_at_5_max value: 16.217756878539337 - type: nauc_map_at_5_std value: -10.714238788014212 - type: nauc_mrr_at_1000_diff1 value: 28.488264933723528 - type: nauc_mrr_at_1000_max value: 45.94151165403325 - type: nauc_mrr_at_1000_std value: 25.20231778025588 - type: nauc_mrr_at_100_diff1 value: 28.4886630218298 - type: nauc_mrr_at_100_max value: 45.9702575916014 - type: nauc_mrr_at_100_std value: 25.22848732842774 - type: nauc_mrr_at_10_diff1 value: 28.535257017998294 - type: nauc_mrr_at_10_max value: 45.86005605851268 - type: nauc_mrr_at_10_std value: 24.81744203643852 - type: nauc_mrr_at_1_diff1 value: 29.824630548327285 - type: nauc_mrr_at_1_max value: 44.19891968145314 - type: nauc_mrr_at_1_std value: 23.21413139777098 - type: nauc_mrr_at_20_diff1 value: 28.54642005356483 - type: nauc_mrr_at_20_max value: 46.08926361963997 - type: nauc_mrr_at_20_std value: 25.39517294920476 - type: nauc_mrr_at_3_diff1 value: 28.230929109259407 - type: nauc_mrr_at_3_max value: 44.05364599618201 - type: nauc_mrr_at_3_std value: 23.828100697992724 - type: nauc_mrr_at_5_diff1 value: 29.669751924690758 - type: nauc_mrr_at_5_max value: 45.36862661497384 - type: nauc_mrr_at_5_std value: 23.787166807022505 - type: nauc_ndcg_at_1000_diff1 value: 18.515898773404377 - type: nauc_ndcg_at_1000_max value: 44.57748675979855 - type: nauc_ndcg_at_1000_std value: 29.205899131269604 - type: nauc_ndcg_at_100_diff1 value: 15.88197701276405 - type: nauc_ndcg_at_100_max value: 39.62665883972109 - type: nauc_ndcg_at_100_std value: 25.186347352251754 - type: nauc_ndcg_at_10_diff1 value: 16.220798038950925 - type: nauc_ndcg_at_10_max value: 39.67757337154769 - type: nauc_ndcg_at_10_std value: 25.634534917262403 - type: nauc_ndcg_at_1_diff1 value: 31.448775879462932 - type: nauc_ndcg_at_1_max value: 44.4256421079556 - type: nauc_ndcg_at_1_std value: 23.093987850437355 - type: nauc_ndcg_at_20_diff1 value: 15.417507391228035 - type: nauc_ndcg_at_20_max value: 37.52014353976055 - type: nauc_ndcg_at_20_std value: 23.880617920537915 - type: nauc_ndcg_at_3_diff1 value: 18.01018470616153 - type: nauc_ndcg_at_3_max value: 39.135814950810804 - type: nauc_ndcg_at_3_std value: 21.40850285781106 - type: nauc_ndcg_at_5_diff1 value: 18.502338826072368 - type: nauc_ndcg_at_5_max value: 40.2043937728194 - type: nauc_ndcg_at_5_std value: 22.242499743433424 - type: nauc_precision_at_1000_diff1 value: -13.648652068964681 - type: nauc_precision_at_1000_max value: 3.5821865423513426 - type: nauc_precision_at_1000_std value: 35.481456041211274 - type: nauc_precision_at_100_diff1 value: -11.342790040792961 - type: nauc_precision_at_100_max value: 18.41811151847882 - type: nauc_precision_at_100_std value: 44.901842372597336 - type: nauc_precision_at_10_diff1 value: -1.9404654865248405 - type: nauc_precision_at_10_max value: 40.91955602631143 - type: nauc_precision_at_10_std value: 41.38128398646734 - type: nauc_precision_at_1_diff1 value: 29.824630548327285 - type: nauc_precision_at_1_max value: 44.19891968145314 - type: nauc_precision_at_1_std value: 23.21413139777098 - type: nauc_precision_at_20_diff1 value: -5.046696327994225 - type: nauc_precision_at_20_max value: 33.653422186725386 - type: nauc_precision_at_20_std value: 40.97689615511939 - type: nauc_precision_at_3_diff1 value: 5.1767717826900785 - type: nauc_precision_at_3_max value: 38.01276130261592 - type: nauc_precision_at_3_std value: 25.71468883159735 - type: nauc_precision_at_5_diff1 value: 3.847065262189492 - type: nauc_precision_at_5_max value: 41.00941977122254 - type: nauc_precision_at_5_std value: 31.044768384177246 - type: nauc_recall_at_1000_diff1 value: 7.975632504947066 - type: nauc_recall_at_1000_max value: 18.83264064904865 - type: nauc_recall_at_1000_std value: 15.023940189337717 - type: nauc_recall_at_100_diff1 value: 10.354458867884487 - type: nauc_recall_at_100_max value: 27.16900376430975 - type: nauc_recall_at_100_std value: 14.160333284050214 - type: nauc_recall_at_10_diff1 value: 18.04347857307359 - type: nauc_recall_at_10_max value: 19.082544744457774 - type: nauc_recall_at_10_std value: -5.107813434157397 - type: nauc_recall_at_1_diff1 value: 37.94611136294878 - type: nauc_recall_at_1_max value: 7.297492349938244 - type: nauc_recall_at_1_std value: -17.813930346562152 - type: nauc_recall_at_20_diff1 value: 16.658153504941193 - type: nauc_recall_at_20_max value: 23.214261213582382 - type: nauc_recall_at_20_std value: -0.6964816170313349 - type: nauc_recall_at_3_diff1 value: 23.65600569767465 - type: nauc_recall_at_3_max value: 6.543906048065431 - type: nauc_recall_at_3_std value: -15.496093666790777 - type: nauc_recall_at_5_diff1 value: 22.112315726267077 - type: nauc_recall_at_5_max value: 12.258969896916307 - type: nauc_recall_at_5_std value: -12.922832334587008 - type: ndcg_at_1 value: 50.929 - type: ndcg_at_10 value: 40.463 - type: ndcg_at_100 value: 36.909 - type: ndcg_at_1000 value: 45.617999999999995 - type: ndcg_at_20 value: 37.772 - type: ndcg_at_3 value: 46.315 - type: ndcg_at_5 value: 44.052 - type: precision_at_1 value: 52.632 - type: precision_at_10 value: 29.814 - type: precision_at_100 value: 9.325 - type: precision_at_1000 value: 2.236 - type: precision_at_20 value: 22.073999999999998 - type: precision_at_3 value: 42.931000000000004 - type: precision_at_5 value: 37.957 - type: recall_at_1 value: 6.4990000000000006 - type: recall_at_10 value: 20.232 - type: recall_at_100 value: 36.846000000000004 - type: recall_at_1000 value: 69.03 - type: recall_at_20 value: 24.448 - type: recall_at_3 value: 13.258000000000001 - type: recall_at_5 value: 16.255 task: type: Retrieval - dataset: config: default name: MTEB NQ revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 split: test type: mteb/nq metrics: - type: main_score value: 70.518 - type: map_at_1 value: 46.233999999999995 - type: map_at_10 value: 63.519999999999996 - type: map_at_100 value: 64.14699999999999 - type: map_at_1000 value: 64.154 - type: map_at_20 value: 63.975 - type: map_at_3 value: 59.797 - type: map_at_5 value: 62.226000000000006 - type: mrr_at_1 value: 51.76709154113557 - type: mrr_at_10 value: 65.79852489470095 - type: mrr_at_100 value: 66.19480681115492 - type: mrr_at_1000 value: 66.19993656063721 - type: mrr_at_20 value: 66.0923632685851 - type: mrr_at_3 value: 63.185592893008746 - type: mrr_at_5 value: 64.93385477018151 - type: nauc_map_at_1000_diff1 value: 43.35077155361084 - type: nauc_map_at_1000_max value: 37.282536180921085 - type: nauc_map_at_1000_std value: -4.64357984773174 - type: nauc_map_at_100_diff1 value: 43.35098576601616 - type: nauc_map_at_100_max value: 37.28998747522813 - type: nauc_map_at_100_std value: -4.638151362399621 - type: nauc_map_at_10_diff1 value: 43.131007214082594 - type: nauc_map_at_10_max value: 37.430076712266846 - type: nauc_map_at_10_std value: -4.90614475410035 - type: nauc_map_at_1_diff1 value: 45.843123692592485 - type: nauc_map_at_1_max value: 30.160164681399227 - type: nauc_map_at_1_std value: -6.110582951655118 - type: nauc_map_at_20_diff1 value: 43.30588135441681 - type: nauc_map_at_20_max value: 37.41321766111187 - type: nauc_map_at_20_std value: -4.628074353861448 - type: nauc_map_at_3_diff1 value: 42.690411835598695 - type: nauc_map_at_3_max value: 36.64069333510947 - type: nauc_map_at_3_std value: -6.2899609993355545 - type: nauc_map_at_5_diff1 value: 42.906814471744134 - type: nauc_map_at_5_max value: 37.27599132551781 - type: nauc_map_at_5_std value: -5.512203849661435 - type: nauc_mrr_at_1000_diff1 value: 43.77989113830799 - type: nauc_mrr_at_1000_max value: 38.01009876981156 - type: nauc_mrr_at_1000_std value: -2.0250764367321654 - type: nauc_mrr_at_100_diff1 value: 43.78071481914773 - type: nauc_mrr_at_100_max value: 38.01603112272088 - type: nauc_mrr_at_100_std value: -2.019685020907906 - type: nauc_mrr_at_10_diff1 value: 43.582338882429156 - type: nauc_mrr_at_10_max value: 38.19577506300954 - type: nauc_mrr_at_10_std value: -2.011905402842086 - type: nauc_mrr_at_1_diff1 value: 46.544635554669576 - type: nauc_mrr_at_1_max value: 33.82720628969995 - type: nauc_mrr_at_1_std value: -2.924293824382781 - type: nauc_mrr_at_20_diff1 value: 43.713682995581614 - type: nauc_mrr_at_20_max value: 38.09918392374771 - type: nauc_mrr_at_20_std value: -1.9583477023239 - type: nauc_mrr_at_3_diff1 value: 43.35807398052401 - type: nauc_mrr_at_3_max value: 38.39129780935902 - type: nauc_mrr_at_3_std value: -2.287791352096624 - type: nauc_mrr_at_5_diff1 value: 43.4126448419642 - type: nauc_mrr_at_5_max value: 38.27294037073721 - type: nauc_mrr_at_5_std value: -2.166655666337289 - type: nauc_ndcg_at_1000_diff1 value: 43.26202839737687 - type: nauc_ndcg_at_1000_max value: 38.493273787010615 - type: nauc_ndcg_at_1000_std value: -2.9983001465713524 - type: nauc_ndcg_at_100_diff1 value: 43.25688556190981 - type: nauc_ndcg_at_100_max value: 38.68155788574137 - type: nauc_ndcg_at_100_std value: -2.8355616191757487 - type: nauc_ndcg_at_10_diff1 value: 42.37071983774907 - type: nauc_ndcg_at_10_max value: 39.60970139451164 - type: nauc_ndcg_at_10_std value: -3.5877671856177775 - type: nauc_ndcg_at_1_diff1 value: 46.614780156517845 - type: nauc_ndcg_at_1_max value: 33.863655999315526 - type: nauc_ndcg_at_1_std value: -2.839239881422542 - type: nauc_ndcg_at_20_diff1 value: 42.97845395193472 - type: nauc_ndcg_at_20_max value: 39.53053589334249 - type: nauc_ndcg_at_20_std value: -2.6507495263904515 - type: nauc_ndcg_at_3_diff1 value: 41.65390869521735 - type: nauc_ndcg_at_3_max value: 38.4851846089685 - type: nauc_ndcg_at_3_std value: -5.6296606018146 - type: nauc_ndcg_at_5_diff1 value: 41.89640848285409 - type: nauc_ndcg_at_5_max value: 39.293659812249615 - type: nauc_ndcg_at_5_std value: -4.754462409312945 - type: nauc_precision_at_1000_diff1 value: -10.848480634403051 - type: nauc_precision_at_1000_max value: 1.3436973699935175 - type: nauc_precision_at_1000_std value: 19.044141500097957 - type: nauc_precision_at_100_diff1 value: -9.018095533261604 - type: nauc_precision_at_100_max value: 4.0402155161025695 - type: nauc_precision_at_100_std value: 19.492823636364996 - type: nauc_precision_at_10_diff1 value: 3.947100636096294 - type: nauc_precision_at_10_max value: 20.598641503195907 - type: nauc_precision_at_10_std value: 13.522240087840858 - type: nauc_precision_at_1_diff1 value: 46.614780156517845 - type: nauc_precision_at_1_max value: 33.863655999315526 - type: nauc_precision_at_1_std value: -2.839239881422542 - type: nauc_precision_at_20_diff1 value: -2.1791072352475336 - type: nauc_precision_at_20_max value: 14.03887841842901 - type: nauc_precision_at_20_std value: 18.846129471001632 - type: nauc_precision_at_3_diff1 value: 21.09092861833543 - type: nauc_precision_at_3_max value: 34.122841034361805 - type: nauc_precision_at_3_std value: 2.5513201020031064 - type: nauc_precision_at_5_diff1 value: 12.181140062410874 - type: nauc_precision_at_5_max value: 27.903435474574234 - type: nauc_precision_at_5_std value: 7.6589638998570315 - type: nauc_recall_at_1000_diff1 value: 59.28482230176634 - type: nauc_recall_at_1000_max value: 85.47306385133284 - type: nauc_recall_at_1000_std value: 76.45740117805659 - type: nauc_recall_at_100_diff1 value: 44.31190730138568 - type: nauc_recall_at_100_max value: 66.30976579719086 - type: nauc_recall_at_100_std value: 30.65274229759539 - type: nauc_recall_at_10_diff1 value: 34.885747244334866 - type: nauc_recall_at_10_max value: 50.998198327439404 - type: nauc_recall_at_10_std value: -2.7025509359838193 - type: nauc_recall_at_1_diff1 value: 45.843123692592485 - type: nauc_recall_at_1_max value: 30.160164681399227 - type: nauc_recall_at_1_std value: -6.110582951655118 - type: nauc_recall_at_20_diff1 value: 37.873054394800825 - type: nauc_recall_at_20_max value: 59.21039923637266 - type: nauc_recall_at_20_std value: 9.352312696050557 - type: nauc_recall_at_3_diff1 value: 35.703271085627776 - type: nauc_recall_at_3_max value: 41.19400688280121 - type: nauc_recall_at_3_std value: -7.9624895195139 - type: nauc_recall_at_5_diff1 value: 34.831972383157925 - type: nauc_recall_at_5_max value: 44.82018386701478 - type: nauc_recall_at_5_std value: -7.046789506164082 - type: ndcg_at_1 value: 51.73799999999999 - type: ndcg_at_10 value: 70.518 - type: ndcg_at_100 value: 72.841 - type: ndcg_at_1000 value: 72.99799999999999 - type: ndcg_at_20 value: 71.895 - type: ndcg_at_3 value: 64.06500000000001 - type: ndcg_at_5 value: 67.86999999999999 - type: precision_at_1 value: 51.73799999999999 - type: precision_at_10 value: 10.698 - type: precision_at_100 value: 1.2 - type: precision_at_1000 value: 0.121 - type: precision_at_20 value: 5.691 - type: precision_at_3 value: 28.37 - type: precision_at_5 value: 19.363 - type: recall_at_1 value: 46.233999999999995 - type: recall_at_10 value: 89.062 - type: recall_at_100 value: 98.622 - type: recall_at_1000 value: 99.754 - type: recall_at_20 value: 94.052 - type: recall_at_3 value: 72.994 - type: recall_at_5 value: 81.525 task: type: Retrieval - dataset: config: default name: MTEB QuoraRetrieval revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 split: test type: mteb/quora metrics: - type: main_score value: 89.885 - type: map_at_1 value: 72.379 - type: map_at_10 value: 86.455 - type: map_at_100 value: 87.087 - type: map_at_1000 value: 87.1 - type: map_at_20 value: 86.883 - type: map_at_3 value: 83.663 - type: map_at_5 value: 85.443 - type: mrr_at_1 value: 83.27 - type: mrr_at_10 value: 89.13586904761888 - type: mrr_at_100 value: 89.22177886254626 - type: mrr_at_1000 value: 89.22204575963424 - type: mrr_at_20 value: 89.20621913458041 - type: mrr_at_3 value: 88.37999999999981 - type: mrr_at_5 value: 88.89349999999978 - type: nauc_map_at_1000_diff1 value: 78.17410401832315 - type: nauc_map_at_1000_max value: 33.114749237960986 - type: nauc_map_at_1000_std value: -49.79724283243796 - type: nauc_map_at_100_diff1 value: 78.17873434671671 - type: nauc_map_at_100_max value: 33.101626543573325 - type: nauc_map_at_100_std value: -49.82883017160494 - type: nauc_map_at_10_diff1 value: 78.28052682172311 - type: nauc_map_at_10_max value: 32.626693803188694 - type: nauc_map_at_10_std value: -51.941057676350034 - type: nauc_map_at_1_diff1 value: 81.06079816824507 - type: nauc_map_at_1_max value: 25.638093235123616 - type: nauc_map_at_1_std value: -43.230210939240344 - type: nauc_map_at_20_diff1 value: 78.22103944842512 - type: nauc_map_at_20_max value: 32.94488423505404 - type: nauc_map_at_20_std value: -50.69181407781227 - type: nauc_map_at_3_diff1 value: 78.75453877967588 - type: nauc_map_at_3_max value: 30.645950847243686 - type: nauc_map_at_3_std value: -52.983886453956266 - type: nauc_map_at_5_diff1 value: 78.44984884302167 - type: nauc_map_at_5_max value: 31.69697839442234 - type: nauc_map_at_5_std value: -53.21480554718401 - type: nauc_mrr_at_1000_diff1 value: 78.90502271071976 - type: nauc_mrr_at_1000_max value: 35.902725888631075 - type: nauc_mrr_at_1000_std value: -45.82579843551156 - type: nauc_mrr_at_100_diff1 value: 78.90552803580407 - type: nauc_mrr_at_100_max value: 35.90392790964254 - type: nauc_mrr_at_100_std value: -45.82489205475015 - type: nauc_mrr_at_10_diff1 value: 78.89432223469271 - type: nauc_mrr_at_10_max value: 35.86669566861425 - type: nauc_mrr_at_10_std value: -46.0616841694464 - type: nauc_mrr_at_1_diff1 value: 79.53513360034344 - type: nauc_mrr_at_1_max value: 35.299514657188006 - type: nauc_mrr_at_1_std value: -43.17936948437256 - type: nauc_mrr_at_20_diff1 value: 78.90707352031835 - type: nauc_mrr_at_20_max value: 35.906499072241296 - type: nauc_mrr_at_20_std value: -45.8904084451193 - type: nauc_mrr_at_3_diff1 value: 78.70913062166218 - type: nauc_mrr_at_3_max value: 36.16709621132144 - type: nauc_mrr_at_3_std value: -46.00948004774822 - type: nauc_mrr_at_5_diff1 value: 78.91095031555673 - type: nauc_mrr_at_5_max value: 36.010878683954566 - type: nauc_mrr_at_5_std value: -46.31731368609175 - type: nauc_ndcg_at_1000_diff1 value: 78.19132492477127 - type: nauc_ndcg_at_1000_max value: 34.5208358892501 - type: nauc_ndcg_at_1000_std value: -47.938360906488974 - type: nauc_ndcg_at_100_diff1 value: 78.24549799575261 - type: nauc_ndcg_at_100_max value: 34.48869025578818 - type: nauc_ndcg_at_100_std value: -48.02996375451253 - type: nauc_ndcg_at_10_diff1 value: 78.15340584208084 - type: nauc_ndcg_at_10_max value: 33.5226981818058 - type: nauc_ndcg_at_10_std value: -51.690477519601494 - type: nauc_ndcg_at_1_diff1 value: 79.55459365767561 - type: nauc_ndcg_at_1_max value: 35.25214101433387 - type: nauc_ndcg_at_1_std value: -43.10088819860409 - type: nauc_ndcg_at_20_diff1 value: 78.27277286768546 - type: nauc_ndcg_at_20_max value: 33.997104745595564 - type: nauc_ndcg_at_20_std value: -50.10549601980995 - type: nauc_ndcg_at_3_diff1 value: 77.68820501917479 - type: nauc_ndcg_at_3_max value: 33.00389630941839 - type: nauc_ndcg_at_3_std value: -51.00595251236665 - type: nauc_ndcg_at_5_diff1 value: 78.08093149961476 - type: nauc_ndcg_at_5_max value: 33.03434664578743 - type: nauc_ndcg_at_5_std value: -52.37122386447497 - type: nauc_precision_at_1000_diff1 value: -44.49830608740945 - type: nauc_precision_at_1000_max value: -7.3283280714307395 - type: nauc_precision_at_1000_std value: 38.55076692876393 - type: nauc_precision_at_100_diff1 value: -44.252675314263904 - type: nauc_precision_at_100_max value: -7.038454433556829 - type: nauc_precision_at_100_std value: 38.247323997481615 - type: nauc_precision_at_10_diff1 value: -40.192852013615216 - type: nauc_precision_at_10_max value: -3.7258976649568036 - type: nauc_precision_at_10_std value: 25.983458444206182 - type: nauc_precision_at_1_diff1 value: 79.55459365767561 - type: nauc_precision_at_1_max value: 35.25214101433387 - type: nauc_precision_at_1_std value: -43.10088819860409 - type: nauc_precision_at_20_diff1 value: -43.020749754821495 - type: nauc_precision_at_20_max value: -5.7062060443801075 - type: nauc_precision_at_20_std value: 32.8862431943092 - type: nauc_precision_at_3_diff1 value: -22.843593386293996 - type: nauc_precision_at_3_max value: 4.474275296763041 - type: nauc_precision_at_3_std value: 6.119920479600398 - type: nauc_precision_at_5_diff1 value: -33.598088334605045 - type: nauc_precision_at_5_max value: -0.41505757559350775 - type: nauc_precision_at_5_std value: 16.52526817965026 - type: nauc_recall_at_1000_diff1 value: 28.726073762912847 - type: nauc_recall_at_1000_max value: -57.390873015654066 - type: nauc_recall_at_1000_std value: 69.71288515421948 - type: nauc_recall_at_100_diff1 value: 83.06070133460443 - type: nauc_recall_at_100_max value: 33.27991294763942 - type: nauc_recall_at_100_std value: -42.785112479889655 - type: nauc_recall_at_10_diff1 value: 74.73877865072825 - type: nauc_recall_at_10_max value: 27.81410621945221 - type: nauc_recall_at_10_std value: -75.85371099008806 - type: nauc_recall_at_1_diff1 value: 81.06079816824507 - type: nauc_recall_at_1_max value: 25.638093235123616 - type: nauc_recall_at_1_std value: -43.230210939240344 - type: nauc_recall_at_20_diff1 value: 76.04615040930837 - type: nauc_recall_at_20_max value: 27.47173316749929 - type: nauc_recall_at_20_std value: -78.29029550423172 - type: nauc_recall_at_3_diff1 value: 75.29987903678384 - type: nauc_recall_at_3_max value: 27.48543826795177 - type: nauc_recall_at_3_std value: -60.91023011356427 - type: nauc_recall_at_5_diff1 value: 74.71682412813378 - type: nauc_recall_at_5_max value: 27.255092143441562 - type: nauc_recall_at_5_std value: -69.03177732393821 - type: ndcg_at_1 value: 83.26 - type: ndcg_at_10 value: 89.885 - type: ndcg_at_100 value: 90.968 - type: ndcg_at_1000 value: 91.02799999999999 - type: ndcg_at_20 value: 90.52900000000001 - type: ndcg_at_3 value: 87.443 - type: ndcg_at_5 value: 88.81 - type: precision_at_1 value: 83.26 - type: precision_at_10 value: 13.581999999999999 - type: precision_at_100 value: 1.541 - type: precision_at_1000 value: 0.157 - type: precision_at_20 value: 7.210999999999999 - type: precision_at_3 value: 38.323 - type: precision_at_5 value: 25.069999999999997 - type: recall_at_1 value: 72.379 - type: recall_at_10 value: 96.261 - type: recall_at_100 value: 99.779 - type: recall_at_1000 value: 99.996 - type: recall_at_20 value: 98.301 - type: recall_at_3 value: 89.101 - type: recall_at_5 value: 93.11500000000001 task: type: Retrieval - dataset: config: default name: MTEB RedditClustering revision: 24640382cdbf8abc73003fb0fa6d111a705499eb split: test type: mteb/reddit-clustering metrics: - type: main_score value: 61.87769077476204 - type: v_measure value: 61.87769077476204 - type: v_measure_std value: 5.290405218730049 task: type: Clustering - dataset: config: default name: MTEB RedditClusteringP2P revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 split: test type: mteb/reddit-clustering-p2p metrics: - type: main_score value: 68.29553057563754 - type: v_measure value: 68.29553057563754 - type: v_measure_std value: 13.019229253711732 task: type: Clustering - dataset: config: default name: MTEB SCIDOCS revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 split: test type: mteb/scidocs metrics: - type: main_score value: 20.86 - type: map_at_1 value: 4.843 - type: map_at_10 value: 12.457 - type: map_at_100 value: 14.648 - type: map_at_1000 value: 14.965 - type: map_at_20 value: 13.596 - type: map_at_3 value: 8.776 - type: map_at_5 value: 10.528 - type: mrr_at_1 value: 23.799999999999997 - type: mrr_at_10 value: 34.93765873015872 - type: mrr_at_100 value: 36.054095036751825 - type: mrr_at_1000 value: 36.10871797082569 - type: mrr_at_20 value: 35.57880859465608 - type: mrr_at_3 value: 31.54999999999999 - type: mrr_at_5 value: 33.53999999999998 - type: nauc_map_at_1000_diff1 value: 16.889540490911525 - type: nauc_map_at_1000_max value: 25.726340275186143 - type: nauc_map_at_1000_std value: 9.926911665196988 - type: nauc_map_at_100_diff1 value: 16.889355521248202 - type: nauc_map_at_100_max value: 25.628741550328126 - type: nauc_map_at_100_std value: 9.637062917997012 - type: nauc_map_at_10_diff1 value: 16.972521218507854 - type: nauc_map_at_10_max value: 24.810172126870363 - type: nauc_map_at_10_std value: 7.09295422867669 - type: nauc_map_at_1_diff1 value: 24.9292922418417 - type: nauc_map_at_1_max value: 15.49253311874767 - type: nauc_map_at_1_std value: -0.4754734108717385 - type: nauc_map_at_20_diff1 value: 16.945564955989113 - type: nauc_map_at_20_max value: 25.197327599885362 - type: nauc_map_at_20_std value: 7.972256233219635 - type: nauc_map_at_3_diff1 value: 19.503723922705067 - type: nauc_map_at_3_max value: 20.795879090480057 - type: nauc_map_at_3_std value: 1.5828913591118658 - type: nauc_map_at_5_diff1 value: 19.80474780705204 - type: nauc_map_at_5_max value: 24.040173591299723 - type: nauc_map_at_5_std value: 4.153642430396917 - type: nauc_mrr_at_1000_diff1 value: 21.80300741603344 - type: nauc_mrr_at_1000_max value: 19.98123409846586 - type: nauc_mrr_at_1000_std value: 3.6325335777371377 - type: nauc_mrr_at_100_diff1 value: 21.804966803578946 - type: nauc_mrr_at_100_max value: 19.9965104601956 - type: nauc_mrr_at_100_std value: 3.6713772865070107 - type: nauc_mrr_at_10_diff1 value: 21.66109150475663 - type: nauc_mrr_at_10_max value: 19.873876575424404 - type: nauc_mrr_at_10_std value: 3.3387503298795584 - type: nauc_mrr_at_1_diff1 value: 24.868548821073084 - type: nauc_mrr_at_1_max value: 16.189915011439044 - type: nauc_mrr_at_1_std value: -0.17692171251799987 - type: nauc_mrr_at_20_diff1 value: 21.677427533247375 - type: nauc_mrr_at_20_max value: 19.967193157614872 - type: nauc_mrr_at_20_std value: 3.639825799332009 - type: nauc_mrr_at_3_diff1 value: 21.681117207511825 - type: nauc_mrr_at_3_max value: 19.132660363303295 - type: nauc_mrr_at_3_std value: 1.6613642176263752 - type: nauc_mrr_at_5_diff1 value: 21.833332207271884 - type: nauc_mrr_at_5_max value: 19.926480855266213 - type: nauc_mrr_at_5_std value: 2.901801717093585 - type: nauc_ndcg_at_1000_diff1 value: 16.92599483752314 - type: nauc_ndcg_at_1000_max value: 27.126582080942814 - type: nauc_ndcg_at_1000_std value: 16.638448489514683 - type: nauc_ndcg_at_100_diff1 value: 16.96586885959473 - type: nauc_ndcg_at_100_max value: 26.675878724175046 - type: nauc_ndcg_at_100_std value: 15.369335585614245 - type: nauc_ndcg_at_10_diff1 value: 16.59779893225997 - type: nauc_ndcg_at_10_max value: 24.865338966132818 - type: nauc_ndcg_at_10_std value: 8.934209252745864 - type: nauc_ndcg_at_1_diff1 value: 24.868548821073084 - type: nauc_ndcg_at_1_max value: 16.189915011439044 - type: nauc_ndcg_at_1_std value: -0.17692171251799987 - type: nauc_ndcg_at_20_diff1 value: 16.647406628819976 - type: nauc_ndcg_at_20_max value: 25.64488140369063 - type: nauc_ndcg_at_20_std value: 10.587157641309098 - type: nauc_ndcg_at_3_diff1 value: 19.093302254257377 - type: nauc_ndcg_at_3_max value: 21.33725971448413 - type: nauc_ndcg_at_3_std value: 2.549021710462978 - type: nauc_ndcg_at_5_diff1 value: 19.495189389728836 - type: nauc_ndcg_at_5_max value: 24.21965138651894 - type: nauc_ndcg_at_5_std value: 5.549408503444251 - type: nauc_precision_at_1000_diff1 value: 7.4232833098081565 - type: nauc_precision_at_1000_max value: 25.24619675919913 - type: nauc_precision_at_1000_std value: 32.79744946411614 - type: nauc_precision_at_100_diff1 value: 10.550449529674747 - type: nauc_precision_at_100_max value: 25.652112631579726 - type: nauc_precision_at_100_std value: 26.65722909800614 - type: nauc_precision_at_10_diff1 value: 11.195653785882708 - type: nauc_precision_at_10_max value: 26.469986306854977 - type: nauc_precision_at_10_std value: 14.05089697514966 - type: nauc_precision_at_1_diff1 value: 24.868548821073084 - type: nauc_precision_at_1_max value: 16.189915011439044 - type: nauc_precision_at_1_std value: -0.17692171251799987 - type: nauc_precision_at_20_diff1 value: 11.16738184991032 - type: nauc_precision_at_20_max value: 26.53741675130711 - type: nauc_precision_at_20_std value: 16.250110771034542 - type: nauc_precision_at_3_diff1 value: 16.917872510926284 - type: nauc_precision_at_3_max value: 23.22094310791854 - type: nauc_precision_at_3_std value: 3.9255078517383906 - type: nauc_precision_at_5_diff1 value: 16.898056883587824 - type: nauc_precision_at_5_max value: 27.39457295203392 - type: nauc_precision_at_5_std value: 8.924759582566171 - type: nauc_recall_at_1000_diff1 value: 7.516072705946253 - type: nauc_recall_at_1000_max value: 25.001682297424594 - type: nauc_recall_at_1000_std value: 33.86296283879721 - type: nauc_recall_at_100_diff1 value: 10.435705067998168 - type: nauc_recall_at_100_max value: 25.31622603650995 - type: nauc_recall_at_100_std value: 26.758897185352097 - type: nauc_recall_at_10_diff1 value: 11.110953419292343 - type: nauc_recall_at_10_max value: 25.970593144433085 - type: nauc_recall_at_10_std value: 13.92252981022314 - type: nauc_recall_at_1_diff1 value: 24.9292922418417 - type: nauc_recall_at_1_max value: 15.49253311874767 - type: nauc_recall_at_1_std value: -0.4754734108717385 - type: nauc_recall_at_20_diff1 value: 11.050515317424548 - type: nauc_recall_at_20_max value: 26.068866115743134 - type: nauc_recall_at_20_std value: 16.13787216291987 - type: nauc_recall_at_3_diff1 value: 17.013383740580203 - type: nauc_recall_at_3_max value: 22.49105285578937 - type: nauc_recall_at_3_std value: 3.5741090487413687 - type: nauc_recall_at_5_diff1 value: 16.973540662242602 - type: nauc_recall_at_5_max value: 26.78087164318061 - type: nauc_recall_at_5_std value: 8.68040862354009 - type: ndcg_at_1 value: 23.799999999999997 - type: ndcg_at_10 value: 20.86 - type: ndcg_at_100 value: 29.145 - type: ndcg_at_1000 value: 34.518 - type: ndcg_at_20 value: 23.892 - type: ndcg_at_3 value: 19.541 - type: ndcg_at_5 value: 17.166999999999998 - type: precision_at_1 value: 23.799999999999997 - type: precision_at_10 value: 10.9 - type: precision_at_100 value: 2.281 - type: precision_at_1000 value: 0.357 - type: precision_at_20 value: 7.21 - type: precision_at_3 value: 18.3 - type: precision_at_5 value: 15.120000000000001 - type: recall_at_1 value: 4.843 - type: recall_at_10 value: 22.12 - type: recall_at_100 value: 46.257 - type: recall_at_1000 value: 72.382 - type: recall_at_20 value: 29.253 - type: recall_at_3 value: 11.158 - type: recall_at_5 value: 15.347 task: type: Retrieval - dataset: config: default name: MTEB SICK-R revision: 20a6d6f312dd54037fe07a32d58e5e168867909d split: test type: mteb/sickr-sts metrics: - type: cosine_pearson value: 85.65467419133347 - type: cosine_spearman value: 81.88046945336663 - type: euclidean_pearson value: 82.82887106181879 - type: euclidean_spearman value: 81.88047605481775 - type: main_score value: 81.88046945336663 - type: manhattan_pearson value: 82.7839019603756 - type: manhattan_spearman value: 81.83505450284663 - type: pearson value: 85.65467419133347 - type: spearman value: 81.88046945336663 task: type: STS - dataset: config: default name: MTEB STS12 revision: a0d554a64d88156834ff5ae9920b964011b16384 split: test type: mteb/sts12-sts metrics: - type: cosine_pearson value: 85.8979872663498 - type: cosine_spearman value: 78.63991285161867 - type: euclidean_pearson value: 81.20243176386163 - type: euclidean_spearman value: 78.64021127260493 - type: main_score value: 78.63991285161867 - type: manhattan_pearson value: 81.58673652635328 - type: manhattan_spearman value: 79.03930665482164 - type: pearson value: 85.8979872663498 - type: spearman value: 78.63991285161867 task: type: STS - dataset: config: default name: MTEB STS13 revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca split: test type: mteb/sts13-sts metrics: - type: cosine_pearson value: 87.10598414063074 - type: cosine_spearman value: 87.12110799581852 - type: euclidean_pearson value: 86.52284239759508 - type: euclidean_spearman value: 87.12110799581852 - type: main_score value: 87.12110799581852 - type: manhattan_pearson value: 86.61105352996736 - type: manhattan_spearman value: 87.34100209521596 - type: pearson value: 87.10598414063074 - type: spearman value: 87.12110799581852 task: type: STS - dataset: config: default name: MTEB STS14 revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 split: test type: mteb/sts14-sts metrics: - type: cosine_pearson value: 85.66540041627184 - type: cosine_spearman value: 83.55263671417923 - type: euclidean_pearson value: 84.2332532036626 - type: euclidean_spearman value: 83.55264421653584 - type: main_score value: 83.55263671417923 - type: manhattan_pearson value: 84.14418954784165 - type: manhattan_spearman value: 83.58193360267302 - type: pearson value: 85.66540041627184 - type: spearman value: 83.55263671417923 task: type: STS - dataset: config: default name: MTEB STS15 revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 split: test type: mteb/sts15-sts metrics: - type: cosine_pearson value: 89.83404956912175 - type: cosine_spearman value: 90.09569633194636 - type: euclidean_pearson value: 89.31121256629982 - type: euclidean_spearman value: 90.09569632193572 - type: main_score value: 90.09569633194636 - type: manhattan_pearson value: 89.30064909066367 - type: manhattan_spearman value: 90.20232732019451 - type: pearson value: 89.83404956912175 - type: spearman value: 90.09569633194636 task: type: STS - dataset: config: default name: MTEB STS16 revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 split: test type: mteb/sts16-sts metrics: - type: cosine_pearson value: 86.27894370732598 - type: cosine_spearman value: 87.22000226558832 - type: euclidean_pearson value: 85.92822715155758 - type: euclidean_spearman value: 87.22000226558832 - type: main_score value: 87.22000226558832 - type: manhattan_pearson value: 85.9498561399522 - type: manhattan_spearman value: 87.28837300894288 - type: pearson value: 86.27894370732598 - type: spearman value: 87.22000226558832 task: type: STS - dataset: config: en-en name: MTEB STS17 (en-en) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 91.60185356782324 - type: cosine_spearman value: 91.43471625912765 - type: euclidean_pearson value: 91.52529087606635 - type: euclidean_spearman value: 91.43471625912765 - type: main_score value: 91.43471625912765 - type: manhattan_pearson value: 91.34917173506308 - type: manhattan_spearman value: 91.2112665439884 - type: pearson value: 91.60185356782324 - type: spearman value: 91.43471625912765 task: type: STS - dataset: config: en name: MTEB STS22 (en) revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 split: test type: mteb/sts22-crosslingual-sts metrics: - type: cosine_pearson value: 68.735098373629 - type: cosine_spearman value: 67.76156085991387 - type: euclidean_pearson value: 68.38053954511516 - type: euclidean_spearman value: 67.76156085991387 - type: main_score value: 67.76156085991387 - type: manhattan_pearson value: 68.4533080173714 - type: manhattan_spearman value: 67.76676959397871 - type: pearson value: 68.735098373629 - type: spearman value: 67.76156085991387 task: type: STS - dataset: config: default name: MTEB STSBenchmark revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 split: test type: mteb/stsbenchmark-sts metrics: - type: cosine_pearson value: 87.63236624274985 - type: cosine_spearman value: 88.27561759951514 - type: euclidean_pearson value: 87.61137355553329 - type: euclidean_spearman value: 88.27561759951514 - type: main_score value: 88.27561759951514 - type: manhattan_pearson value: 87.63505381780153 - type: manhattan_spearman value: 88.41268943146845 - type: pearson value: 87.63236624274985 - type: spearman value: 88.27561759951514 task: type: STS - dataset: config: default name: MTEB SciDocsRR revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab split: test type: mteb/scidocs-reranking metrics: - type: main_score value: 85.16412972900244 - type: map value: 85.16412972900244 - type: mrr value: 96.15786628041529 - type: nAUC_map_diff1 value: -1.5068306084088756 - type: nAUC_map_max value: 48.81296049442589 - type: nAUC_map_std value: 65.0187132933644 - type: nAUC_mrr_diff1 value: 44.22872564939586 - type: nAUC_mrr_max value: 85.19719227096536 - type: nAUC_mrr_std value: 79.62669870868876 task: type: Reranking - dataset: config: default name: MTEB SciFact revision: 0228b52cf27578f30900b9e5271d331663a030d7 split: test type: mteb/scifact metrics: - type: main_score value: 77.377 - type: map_at_1 value: 62.217 - type: map_at_10 value: 73.115 - type: map_at_100 value: 73.63499999999999 - type: map_at_1000 value: 73.644 - type: map_at_20 value: 73.528 - type: map_at_3 value: 70.62 - type: map_at_5 value: 72.16 - type: mrr_at_1 value: 65.0 - type: mrr_at_10 value: 74.05542328042326 - type: mrr_at_100 value: 74.46295785951277 - type: mrr_at_1000 value: 74.47168088874803 - type: mrr_at_20 value: 74.35632423132421 - type: mrr_at_3 value: 72.55555555555556 - type: mrr_at_5 value: 73.38888888888887 - type: nauc_map_at_1000_diff1 value: 72.31754010838618 - type: nauc_map_at_1000_max value: 60.59518156728312 - type: nauc_map_at_1000_std value: -3.601504782295705 - type: nauc_map_at_100_diff1 value: 72.32057771059107 - type: nauc_map_at_100_max value: 60.60481879601873 - type: nauc_map_at_100_std value: -3.6030430073837167 - type: nauc_map_at_10_diff1 value: 72.15009895006031 - type: nauc_map_at_10_max value: 60.49958178006608 - type: nauc_map_at_10_std value: -4.305475753173601 - type: nauc_map_at_1_diff1 value: 76.32919417574946 - type: nauc_map_at_1_max value: 54.77358788281581 - type: nauc_map_at_1_std value: -9.773898055794557 - type: nauc_map_at_20_diff1 value: 72.15740734516393 - type: nauc_map_at_20_max value: 60.61318821265446 - type: nauc_map_at_20_std value: -3.6016854193910803 - type: nauc_map_at_3_diff1 value: 72.07435404889445 - type: nauc_map_at_3_max value: 56.93970890047747 - type: nauc_map_at_3_std value: -8.697324220121793 - type: nauc_map_at_5_diff1 value: 72.42599960854554 - type: nauc_map_at_5_max value: 60.12535137001906 - type: nauc_map_at_5_std value: -4.437892354037166 - type: nauc_mrr_at_1000_diff1 value: 72.75103842052889 - type: nauc_mrr_at_1000_max value: 62.72341811793062 - type: nauc_mrr_at_1000_std value: -0.7759889099766357 - type: nauc_mrr_at_100_diff1 value: 72.75396801842608 - type: nauc_mrr_at_100_max value: 62.73241247525427 - type: nauc_mrr_at_100_std value: -0.7786866224468205 - type: nauc_mrr_at_10_diff1 value: 72.5942754009733 - type: nauc_mrr_at_10_max value: 62.895066542256664 - type: nauc_mrr_at_10_std value: -0.9018200301159104 - type: nauc_mrr_at_1_diff1 value: 77.63311362465076 - type: nauc_mrr_at_1_max value: 62.42059294219759 - type: nauc_mrr_at_1_std value: -1.3182520953698476 - type: nauc_mrr_at_20_diff1 value: 72.58522880943326 - type: nauc_mrr_at_20_max value: 62.73063935403417 - type: nauc_mrr_at_20_std value: -0.7910003366564456 - type: nauc_mrr_at_3_diff1 value: 72.70751722757556 - type: nauc_mrr_at_3_max value: 62.38218933726893 - type: nauc_mrr_at_3_std value: -1.7126398606397155 - type: nauc_mrr_at_5_diff1 value: 72.57550761997256 - type: nauc_mrr_at_5_max value: 62.70945847818393 - type: nauc_mrr_at_5_std value: -0.30886077098332143 - type: nauc_ndcg_at_1000_diff1 value: 71.6036105202873 - type: nauc_ndcg_at_1000_max value: 61.99911514670603 - type: nauc_ndcg_at_1000_std value: -2.050470755577302 - type: nauc_ndcg_at_100_diff1 value: 71.70345074974581 - type: nauc_ndcg_at_100_max value: 62.374525611545714 - type: nauc_ndcg_at_100_std value: -1.922345118135967 - type: nauc_ndcg_at_10_diff1 value: 70.40027928749286 - type: nauc_ndcg_at_10_max value: 62.36595526966657 - type: nauc_ndcg_at_10_std value: -3.862278807246422 - type: nauc_ndcg_at_1_diff1 value: 77.63311362465076 - type: nauc_ndcg_at_1_max value: 62.42059294219759 - type: nauc_ndcg_at_1_std value: -1.3182520953698476 - type: nauc_ndcg_at_20_diff1 value: 70.21719291674641 - type: nauc_ndcg_at_20_max value: 62.356711760569404 - type: nauc_ndcg_at_20_std value: -2.240360396463778 - type: nauc_ndcg_at_3_diff1 value: 70.72483260039468 - type: nauc_ndcg_at_3_max value: 59.465348910073445 - type: nauc_ndcg_at_3_std value: -6.379991854598364 - type: nauc_ndcg_at_5_diff1 value: 70.91296936044013 - type: nauc_ndcg_at_5_max value: 61.5986283773017 - type: nauc_ndcg_at_5_std value: -3.064893399445654 - type: nauc_precision_at_1000_diff1 value: -25.399544557043956 - type: nauc_precision_at_1000_max value: 17.838641101318792 - type: nauc_precision_at_1000_std value: 54.531382221213185 - type: nauc_precision_at_100_diff1 value: -15.78139909072201 - type: nauc_precision_at_100_max value: 24.183801380755472 - type: nauc_precision_at_100_std value: 50.39320972640593 - type: nauc_precision_at_10_diff1 value: 4.1199958514831 - type: nauc_precision_at_10_max value: 37.922630159717926 - type: nauc_precision_at_10_std value: 32.94959551960178 - type: nauc_precision_at_1_diff1 value: 77.63311362465076 - type: nauc_precision_at_1_max value: 62.42059294219759 - type: nauc_precision_at_1_std value: -1.3182520953698476 - type: nauc_precision_at_20_diff1 value: -8.926047159112303 - type: nauc_precision_at_20_max value: 29.369903951067172 - type: nauc_precision_at_20_std value: 41.793379234725904 - type: nauc_precision_at_3_diff1 value: 36.51209832895358 - type: nauc_precision_at_3_max value: 51.07398992745159 - type: nauc_precision_at_3_std value: 13.831661495933623 - type: nauc_precision_at_5_diff1 value: 19.526084047733807 - type: nauc_precision_at_5_max value: 46.67537950098273 - type: nauc_precision_at_5_std value: 31.06747779005178 - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_100_diff1 value: 77.45764972655711 - type: nauc_recall_at_100_max value: 85.69427771108462 - type: nauc_recall_at_100_std value: 10.277444311057575 - type: nauc_recall_at_10_diff1 value: 59.14685653975806 - type: nauc_recall_at_10_max value: 67.75739956082005 - type: nauc_recall_at_10_std value: -12.22251646924215 - type: nauc_recall_at_1_diff1 value: 76.32919417574946 - type: nauc_recall_at_1_max value: 54.77358788281581 - type: nauc_recall_at_1_std value: -9.773898055794557 - type: nauc_recall_at_20_diff1 value: 49.90908644159423 - type: nauc_recall_at_20_max value: 70.55383556931541 - type: nauc_recall_at_20_std value: -3.7004275394368182 - type: nauc_recall_at_3_diff1 value: 64.34183819693267 - type: nauc_recall_at_3_max value: 55.782789721196444 - type: nauc_recall_at_3_std value: -13.886583892174077 - type: nauc_recall_at_5_diff1 value: 63.467364565196135 - type: nauc_recall_at_5_max value: 62.51562390716315 - type: nauc_recall_at_5_std value: -4.715416491952255 - type: ndcg_at_1 value: 65.0 - type: ndcg_at_10 value: 77.377 - type: ndcg_at_100 value: 79.36 - type: ndcg_at_1000 value: 79.644 - type: ndcg_at_20 value: 78.61200000000001 - type: ndcg_at_3 value: 73.624 - type: ndcg_at_5 value: 75.458 - type: precision_at_1 value: 65.0 - type: precision_at_10 value: 10.100000000000001 - type: precision_at_100 value: 1.107 - type: precision_at_1000 value: 0.11299999999999999 - type: precision_at_20 value: 5.333 - type: precision_at_3 value: 28.778 - type: precision_at_5 value: 18.8 - type: recall_at_1 value: 62.217 - type: recall_at_10 value: 89.156 - type: recall_at_100 value: 97.667 - type: recall_at_1000 value: 100.0 - type: recall_at_20 value: 93.667 - type: recall_at_3 value: 79.183 - type: recall_at_5 value: 83.672 task: type: Retrieval - dataset: config: default name: MTEB SprintDuplicateQuestions revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 split: test type: mteb/sprintduplicatequestions-pairclassification metrics: - type: cos_sim_accuracy value: 99.8019801980198 - type: cos_sim_ap value: 95.25139396923107 - type: dot_sim_accuracy value: 0.998019801980198 - type: dot_sim_ap value: 95.25139396923107 - type: max_accuracy value: 99.8019801980198 - type: max_ap value: 95.43878917155146 - type: max_f1 value: 90.0398406374502 - type: main_score value: 95.43878917155146 task: type: PairClassification - dataset: config: default name: MTEB StackExchangeClustering revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 split: test type: mteb/stackexchange-clustering metrics: - type: main_score value: 75.91311883393888 - type: v_measure value: 75.91311883393888 - type: v_measure_std value: 3.286198100593212 task: type: Clustering - dataset: config: default name: MTEB StackExchangeClusteringP2P revision: 815ca46b2622cec33ccafc3735d572c266efdb44 split: test type: mteb/stackexchange-clustering-p2p metrics: - type: main_score value: 47.171049215275694 - type: v_measure value: 47.171049215275694 - type: v_measure_std value: 1.6586563477857534 task: type: Clustering - dataset: config: default name: MTEB StackOverflowDupQuestions revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 split: test type: mteb/stackoverflowdupquestions-reranking metrics: - type: main_score value: 54.15041943470163 - type: map value: 54.15041943470163 - type: mrr value: 55.03112798149563 - type: nAUC_map_diff1 value: 39.50144777017669 - type: nAUC_map_max value: 14.024793174481395 - type: nAUC_map_std value: 6.533766502190137 - type: nAUC_mrr_diff1 value: 39.72560651870919 - type: nAUC_mrr_max value: 14.807887392821616 - type: nAUC_mrr_std value: 7.270272018791473 task: type: Reranking - dataset: config: default name: MTEB SummEval revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c split: test type: mteb/summeval metrics: - type: cosine_pearson value: 31.57515534177576 - type: cosine_spearman value: 31.415247541636194 - type: dot_pearson value: 31.575170220667488 - type: dot_spearman value: 31.415247541636194 - type: main_score value: 31.415247541636194 - type: pearson value: 31.57515534177576 - type: spearman value: 31.415247541636194 task: type: Summarization - dataset: config: default name: MTEB TRECCOVID revision: bb9466bac8153a0349341eb1b22e06409e78ef4e split: test type: mteb/trec-covid metrics: - type: main_score value: 83.67999999999999 - type: map_at_1 value: 0.243 - type: map_at_10 value: 2.167 - type: map_at_100 value: 13.750000000000002 - type: map_at_1000 value: 33.537 - type: map_at_20 value: 4.047 - type: map_at_3 value: 0.694 - type: map_at_5 value: 1.141 - type: mrr_at_1 value: 94.0 - type: mrr_at_10 value: 97.0 - type: mrr_at_100 value: 97.0 - type: mrr_at_1000 value: 97.0 - type: mrr_at_20 value: 97.0 - type: mrr_at_3 value: 97.0 - type: mrr_at_5 value: 97.0 - type: nauc_map_at_1000_diff1 value: 4.890354942949616 - type: nauc_map_at_1000_max value: 29.279958833328408 - type: nauc_map_at_1000_std value: 77.2405348865942 - type: nauc_map_at_100_diff1 value: 31.835069149380868 - type: nauc_map_at_100_max value: 14.523120708509271 - type: nauc_map_at_100_std value: 39.682149025882886 - type: nauc_map_at_10_diff1 value: 43.45574726953753 - type: nauc_map_at_10_max value: -2.9143965183484246 - type: nauc_map_at_10_std value: 2.8052301238653756 - type: nauc_map_at_1_diff1 value: 26.134637426782753 - type: nauc_map_at_1_max value: -3.108959317897608 - type: nauc_map_at_1_std value: -5.781123480253076 - type: nauc_map_at_20_diff1 value: 45.735224340099236 - type: nauc_map_at_20_max value: -1.099022132339708 - type: nauc_map_at_20_std value: 7.6378546013151905 - type: nauc_map_at_3_diff1 value: 35.70649469812688 - type: nauc_map_at_3_max value: -9.710213033638656 - type: nauc_map_at_3_std value: -3.6668161574691056 - type: nauc_map_at_5_diff1 value: 37.6110093992781 - type: nauc_map_at_5_max value: -8.6295080300384 - type: nauc_map_at_5_std value: -3.2709712613287145 - type: nauc_mrr_at_1000_diff1 value: -22.362278244631675 - type: nauc_mrr_at_1000_max value: 63.74105197634592 - type: nauc_mrr_at_1000_std value: 69.88795518207282 - type: nauc_mrr_at_100_diff1 value: -22.362278244631675 - type: nauc_mrr_at_100_max value: 63.74105197634592 - type: nauc_mrr_at_100_std value: 69.88795518207282 - type: nauc_mrr_at_10_diff1 value: -22.362278244631675 - type: nauc_mrr_at_10_max value: 63.74105197634592 - type: nauc_mrr_at_10_std value: 69.88795518207282 - type: nauc_mrr_at_1_diff1 value: -22.36227824463097 - type: nauc_mrr_at_1_max value: 63.741051976346206 - type: nauc_mrr_at_1_std value: 69.88795518207289 - type: nauc_mrr_at_20_diff1 value: -22.362278244631675 - type: nauc_mrr_at_20_max value: 63.74105197634592 - type: nauc_mrr_at_20_std value: 69.88795518207282 - type: nauc_mrr_at_3_diff1 value: -22.362278244631675 - type: nauc_mrr_at_3_max value: 63.74105197634592 - type: nauc_mrr_at_3_std value: 69.88795518207282 - type: nauc_mrr_at_5_diff1 value: -22.362278244631675 - type: nauc_mrr_at_5_max value: 63.74105197634592 - type: nauc_mrr_at_5_std value: 69.88795518207282 - type: nauc_ndcg_at_1000_diff1 value: 11.950362559089744 - type: nauc_ndcg_at_1000_max value: 27.0707842379056 - type: nauc_ndcg_at_1000_std value: 72.43903405163071 - type: nauc_ndcg_at_100_diff1 value: -3.597031398660954 - type: nauc_ndcg_at_100_max value: 24.415981061123944 - type: nauc_ndcg_at_100_std value: 74.01146007854192 - type: nauc_ndcg_at_10_diff1 value: 17.368676394860337 - type: nauc_ndcg_at_10_max value: 27.014276985741652 - type: nauc_ndcg_at_10_std value: 50.032884783457476 - type: nauc_ndcg_at_1_diff1 value: 5.824544582933801 - type: nauc_ndcg_at_1_max value: 39.22818791946299 - type: nauc_ndcg_at_1_std value: 29.32406519654831 - type: nauc_ndcg_at_20_diff1 value: 17.816409720909615 - type: nauc_ndcg_at_20_max value: 25.056392180259827 - type: nauc_ndcg_at_20_std value: 58.05680238138826 - type: nauc_ndcg_at_3_diff1 value: 15.010486876001556 - type: nauc_ndcg_at_3_max value: 4.023535837214374 - type: nauc_ndcg_at_3_std value: 22.55308565809234 - type: nauc_ndcg_at_5_diff1 value: 12.73162605923733 - type: nauc_ndcg_at_5_max value: 15.425379568695105 - type: nauc_ndcg_at_5_std value: 34.4442400670659 - type: nauc_precision_at_1000_diff1 value: -29.218427320110436 - type: nauc_precision_at_1000_max value: 29.90719259849769 - type: nauc_precision_at_1000_std value: 48.95093300052051 - type: nauc_precision_at_100_diff1 value: -6.881054858812464 - type: nauc_precision_at_100_max value: 30.388273677316956 - type: nauc_precision_at_100_std value: 76.1031398803066 - type: nauc_precision_at_10_diff1 value: 24.298416597687574 - type: nauc_precision_at_10_max value: 44.38332754799598 - type: nauc_precision_at_10_std value: 61.64143369558439 - type: nauc_precision_at_1_diff1 value: -22.36227824463097 - type: nauc_precision_at_1_max value: 63.741051976346206 - type: nauc_precision_at_1_std value: 69.88795518207289 - type: nauc_precision_at_20_diff1 value: 21.823848783430545 - type: nauc_precision_at_20_max value: 32.815202091292875 - type: nauc_precision_at_20_std value: 61.4003619545546 - type: nauc_precision_at_3_diff1 value: 7.264709295578332 - type: nauc_precision_at_3_max value: 18.088275115082432 - type: nauc_precision_at_3_std value: 46.315423001044266 - type: nauc_precision_at_5_diff1 value: 19.4281378539196 - type: nauc_precision_at_5_max value: 30.042729922926426 - type: nauc_precision_at_5_std value: 48.803961503134936 - type: nauc_recall_at_1000_diff1 value: 14.078781719704242 - type: nauc_recall_at_1000_max value: 24.205288710944746 - type: nauc_recall_at_1000_std value: 60.19521883992679 - type: nauc_recall_at_100_diff1 value: 34.68620796161708 - type: nauc_recall_at_100_max value: 5.862669275470962 - type: nauc_recall_at_100_std value: 23.779387105339538 - type: nauc_recall_at_10_diff1 value: 41.60859491145645 - type: nauc_recall_at_10_max value: -6.060553984265031 - type: nauc_recall_at_10_std value: -3.0401474174665597 - type: nauc_recall_at_1_diff1 value: 26.134637426782753 - type: nauc_recall_at_1_max value: -3.108959317897608 - type: nauc_recall_at_1_std value: -5.781123480253076 - type: nauc_recall_at_20_diff1 value: 43.884440668985256 - type: nauc_recall_at_20_max value: -5.215456096089841 - type: nauc_recall_at_20_std value: 0.6346955652816175 - type: nauc_recall_at_3_diff1 value: 36.682959590903515 - type: nauc_recall_at_3_max value: -14.003318698999372 - type: nauc_recall_at_3_std value: -8.732791648435722 - type: nauc_recall_at_5_diff1 value: 37.55874033777468 - type: nauc_recall_at_5_max value: -11.475194910000303 - type: nauc_recall_at_5_std value: -8.24171387960509 - type: ndcg_at_1 value: 90.0 - type: ndcg_at_10 value: 83.67999999999999 - type: ndcg_at_100 value: 66.268 - type: ndcg_at_1000 value: 59.95700000000001 - type: ndcg_at_20 value: 80.41199999999999 - type: ndcg_at_3 value: 86.989 - type: ndcg_at_5 value: 85.60600000000001 - type: precision_at_1 value: 94.0 - type: precision_at_10 value: 87.0 - type: precision_at_100 value: 68.10000000000001 - type: precision_at_1000 value: 26.404 - type: precision_at_20 value: 83.7 - type: precision_at_3 value: 91.333 - type: precision_at_5 value: 89.60000000000001 - type: recall_at_1 value: 0.243 - type: recall_at_10 value: 2.307 - type: recall_at_100 value: 16.713 - type: recall_at_1000 value: 56.433 - type: recall_at_20 value: 4.3950000000000005 - type: recall_at_3 value: 0.721 - type: recall_at_5 value: 1.194 task: type: Retrieval - dataset: config: default name: MTEB Touche2020 revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f split: test type: mteb/touche2020 metrics: - type: main_score value: 27.095999999999997 - type: map_at_1 value: 2.708 - type: map_at_10 value: 10.926 - type: map_at_100 value: 17.023 - type: map_at_1000 value: 18.802 - type: map_at_20 value: 14.075 - type: map_at_3 value: 6.213 - type: map_at_5 value: 8.399 - type: mrr_at_1 value: 38.775510204081634 - type: mrr_at_10 value: 55.96938775510204 - type: mrr_at_100 value: 56.566806209663355 - type: mrr_at_1000 value: 56.586429443572314 - type: mrr_at_20 value: 56.566806209663355 - type: mrr_at_3 value: 53.06122448979592 - type: mrr_at_5 value: 54.48979591836734 - type: nauc_map_at_1000_diff1 value: -3.7447722422200505 - type: nauc_map_at_1000_max value: -21.154942580599432 - type: nauc_map_at_1000_std value: -3.3769353126366854 - type: nauc_map_at_100_diff1 value: -4.211734469956019 - type: nauc_map_at_100_max value: -20.97390043676955 - type: nauc_map_at_100_std value: -7.108253122379712 - type: nauc_map_at_10_diff1 value: -2.503617891657346 - type: nauc_map_at_10_max value: -19.76603379959943 - type: nauc_map_at_10_std value: -24.813694071646186 - type: nauc_map_at_1_diff1 value: -0.6946291628228135 - type: nauc_map_at_1_max value: -27.928780525228326 - type: nauc_map_at_1_std value: -26.644256007057386 - type: nauc_map_at_20_diff1 value: -8.140689983350077 - type: nauc_map_at_20_max value: -21.331762857202346 - type: nauc_map_at_20_std value: -18.46503512945984 - type: nauc_map_at_3_diff1 value: 1.4806459479634606 - type: nauc_map_at_3_max value: -20.57096686541149 - type: nauc_map_at_3_std value: -27.53855079505183 - type: nauc_map_at_5_diff1 value: -2.4911557022868833 - type: nauc_map_at_5_max value: -18.468614237544944 - type: nauc_map_at_5_std value: -27.422000270609885 - type: nauc_mrr_at_1000_diff1 value: 0.5901324153382446 - type: nauc_mrr_at_1000_max value: -29.43201840557888 - type: nauc_mrr_at_1000_std value: -22.113570283308878 - type: nauc_mrr_at_100_diff1 value: 0.6140852308779037 - type: nauc_mrr_at_100_max value: -29.423158073762274 - type: nauc_mrr_at_100_std value: -22.03830742373018 - type: nauc_mrr_at_10_diff1 value: 1.4017303142295798 - type: nauc_mrr_at_10_max value: -29.96128226635445 - type: nauc_mrr_at_10_std value: -21.182800337655188 - type: nauc_mrr_at_1_diff1 value: 2.9967188734445642 - type: nauc_mrr_at_1_max value: -28.076201809234135 - type: nauc_mrr_at_1_std value: -23.829475793931397 - type: nauc_mrr_at_20_diff1 value: 0.6140852308779037 - type: nauc_mrr_at_20_max value: -29.423158073762274 - type: nauc_mrr_at_20_std value: -22.03830742373018 - type: nauc_mrr_at_3_diff1 value: -1.7324100961545983 - type: nauc_mrr_at_3_max value: -31.25504536750873 - type: nauc_mrr_at_3_std value: -27.693245095141595 - type: nauc_mrr_at_5_diff1 value: 0.9366378266246876 - type: nauc_mrr_at_5_max value: -28.61911855691654 - type: nauc_mrr_at_5_std value: -23.51734198003236 - type: nauc_ndcg_at_1000_diff1 value: 5.589806586986813 - type: nauc_ndcg_at_1000_max value: -25.54091728191453 - type: nauc_ndcg_at_1000_std value: 18.867289766624364 - type: nauc_ndcg_at_100_diff1 value: 5.269555604924481 - type: nauc_ndcg_at_100_max value: -25.294068947248 - type: nauc_ndcg_at_100_std value: 12.57359579076201 - type: nauc_ndcg_at_10_diff1 value: -1.8036041625138828 - type: nauc_ndcg_at_10_max value: -23.89433650527811 - type: nauc_ndcg_at_10_std value: -18.669805340174104 - type: nauc_ndcg_at_1_diff1 value: 1.7320960153524356 - type: nauc_ndcg_at_1_max value: -30.98970297820504 - type: nauc_ndcg_at_1_std value: -22.039818727732 - type: nauc_ndcg_at_20_diff1 value: -7.71266194406333 - type: nauc_ndcg_at_20_max value: -28.764052281890564 - type: nauc_ndcg_at_20_std value: -14.058766573885803 - type: nauc_ndcg_at_3_diff1 value: 3.4222049394447023 - type: nauc_ndcg_at_3_max value: -23.010397388596147 - type: nauc_ndcg_at_3_std value: -23.917570461776442 - type: nauc_ndcg_at_5_diff1 value: 0.4359085390014115 - type: nauc_ndcg_at_5_max value: -18.328017574440583 - type: nauc_ndcg_at_5_std value: -22.301590122411703 - type: nauc_precision_at_1000_diff1 value: 5.705380133328601 - type: nauc_precision_at_1000_max value: 29.738757046781583 - type: nauc_precision_at_1000_std value: 37.25317043193516 - type: nauc_precision_at_100_diff1 value: 18.099479915822755 - type: nauc_precision_at_100_max value: 1.039647603335084 - type: nauc_precision_at_100_std value: 68.43506311503532 - type: nauc_precision_at_10_diff1 value: 1.6010906915801002 - type: nauc_precision_at_10_max value: -16.21198992516715 - type: nauc_precision_at_10_std value: -10.55666484527 - type: nauc_precision_at_1_diff1 value: 2.9967188734445642 - type: nauc_precision_at_1_max value: -28.076201809234135 - type: nauc_precision_at_1_std value: -23.829475793931397 - type: nauc_precision_at_20_diff1 value: -9.646266503089361 - type: nauc_precision_at_20_max value: -19.25399592456934 - type: nauc_precision_at_20_std value: 4.154373672246843 - type: nauc_precision_at_3_diff1 value: 6.468923962729313 - type: nauc_precision_at_3_max value: -16.75495139962792 - type: nauc_precision_at_3_std value: -24.1555216494731 - type: nauc_precision_at_5_diff1 value: 1.89724542441865 - type: nauc_precision_at_5_max value: -10.916266272968988 - type: nauc_precision_at_5_std value: -19.996228467499165 - type: nauc_recall_at_1000_diff1 value: -0.3248897031222208 - type: nauc_recall_at_1000_max value: -25.08629526651275 - type: nauc_recall_at_1000_std value: 72.42326605733102 - type: nauc_recall_at_100_diff1 value: 0.20011224230233096 - type: nauc_recall_at_100_max value: -25.71382782994985 - type: nauc_recall_at_100_std value: 31.40559917674001 - type: nauc_recall_at_10_diff1 value: -7.502107897824034 - type: nauc_recall_at_10_max value: -26.197156105779833 - type: nauc_recall_at_10_std value: -20.067019662396106 - type: nauc_recall_at_1_diff1 value: -0.6946291628228135 - type: nauc_recall_at_1_max value: -27.928780525228326 - type: nauc_recall_at_1_std value: -26.644256007057386 - type: nauc_recall_at_20_diff1 value: -16.829462200879107 - type: nauc_recall_at_20_max value: -29.55978083865099 - type: nauc_recall_at_20_std value: -11.329177422867945 - type: nauc_recall_at_3_diff1 value: -4.487251181022699 - type: nauc_recall_at_3_max value: -26.28852595660599 - type: nauc_recall_at_3_std value: -30.010933869743877 - type: nauc_recall_at_5_diff1 value: -7.4729339604681515 - type: nauc_recall_at_5_max value: -22.995431038489112 - type: nauc_recall_at_5_std value: -27.623494423158906 - type: ndcg_at_1 value: 35.714 - type: ndcg_at_10 value: 27.095999999999997 - type: ndcg_at_100 value: 37.577 - type: ndcg_at_1000 value: 50.234 - type: ndcg_at_20 value: 28.706 - type: ndcg_at_3 value: 34.808 - type: ndcg_at_5 value: 31.657999999999998 - type: precision_at_1 value: 38.775999999999996 - type: precision_at_10 value: 23.061 - type: precision_at_100 value: 7.388 - type: precision_at_1000 value: 1.5650000000000002 - type: precision_at_20 value: 18.776 - type: precision_at_3 value: 36.735 - type: precision_at_5 value: 31.429000000000002 - type: recall_at_1 value: 2.708 - type: recall_at_10 value: 16.645 - type: recall_at_100 value: 45.953 - type: recall_at_1000 value: 84.553 - type: recall_at_20 value: 26.259 - type: recall_at_3 value: 7.869 - type: recall_at_5 value: 11.166 task: type: Retrieval - dataset: config: default name: MTEB ToxicConversationsClassification revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de split: test type: mteb/toxic_conversations_50k metrics: - type: accuracy value: 75.87890625 - type: ap value: 16.4629793865873 - type: ap_weighted value: 16.4629793865873 - type: f1 value: 58.32993265544471 - type: f1_weighted value: 80.94360012442658 - type: main_score value: 75.87890625 task: type: Classification - dataset: config: default name: MTEB TweetSentimentExtractionClassification revision: d604517c81ca91fe16a244d1248fc021f9ecee7a split: test type: mteb/tweet_sentiment_extraction metrics: - type: accuracy value: 65.21788341822298 - type: f1 value: 65.00914562845475 - type: f1_weighted value: 63.672388825903845 - type: main_score value: 65.21788341822298 task: type: Classification - dataset: config: default name: MTEB TwentyNewsgroupsClustering revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 split: test type: mteb/twentynewsgroups-clustering metrics: - type: main_score value: 57.152337838073485 - type: v_measure value: 57.152337838073485 - type: v_measure_std value: 0.8799366494028795 task: type: Clustering - dataset: config: default name: MTEB TwitterSemEval2015 revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 split: test type: mteb/twittersemeval2015-pairclassification metrics: - type: max_accuracy value: 88.63920844012637 - type: max_ap value: 81.41444048232692 - type: max_f1 value: 74.84396892115653 - type: accuracy value: 88.63920844012637 - type: accuracy_threshold value: 84.5294713973999 - type: ap value: 81.41443623323144 - type: f1 value: 74.84396892115653 - type: f1_threshold value: 82.87262320518494 - type: precision value: 72.34671263235656 - type: recall value: 77.5197889182058 - type: main_score value: 81.41444048232692 task: type: PairClassification - dataset: config: default name: MTEB TwitterURLCorpus revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf split: test type: mteb/twitterurlcorpus-pairclassification metrics: - type: max_accuracy value: 89.90569332867622 - type: max_ap value: 87.91825594329686 - type: max_f1 value: 80.35081439054949 - type: accuracy value: 89.90569332867622 - type: accuracy_threshold value: 81.01733326911926 - type: ap value: 87.91824445175028 - type: f1 value: 80.35081439054949 - type: f1_threshold value: 78.65387201309204 - type: precision value: 75.0853013982739 - type: recall value: 86.41053279950724 - type: main_score value: 87.91825594329686 task: type: PairClassification tags: - mteb - transformers - sentence-transformers --- ## Zeta-Alpha-E5-Mistral We introduce Zeta Alpha's first public embedding model, a retrieval-specialized, 7B parameter embedding model trained on top of [E5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct). This model marks the first published model from Zeta Alpha's open science embedding models. Check out our blog post for a complete breakdown of the training set we used and all the training details: [Zeta Alpha blog](https://www.zeta-alpha.com/post/fine-tuning-an-llm-for-state-of-the-art-retrieval-zeta-alpha-s-top-10-submission-to-the-the-mteb-be) We are also making available our internal evaluation set, called [NanoBEIR](https://huggingface.co/collections/zeta-alpha-ai/nanobeir-66e1a0af21dfd93e620cd9f6), a collection of Nano (i.e., 50 queries+~10k documents) per BEIR dataset. ### Lora Weights The lora weights are also available, so there is no need to download the full model. ## How to Run The model was trained with the same instruction-tuning strategy as the original [E5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct) model. Therefore, queries should be formatted as follows: ``` Instruct: \nQuery: ``` ### Sentence Transformers ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("zeta-alpha-ai/Zeta-Alpha-E5-Mistral") def get_detailed_instruct(task_description: str, query: str) -> str: return f'Instruct: {task_description}\nQuery: {query}' task = "Given a claim about climate change, retrieve documents that support or refute the claim" queries = [ get_detailed_instruct(task, "In Alaska, brown bears are changing their feeding habits to eat elderberries that ripen earlier."), get_detailed_instruct(task, "Local and regional sea levels continue to exhibit typical natural variability—in some places rising and in others falling.") ] passages = [ "The brown bear ( Ursus arctos ) is a large bear with the widest distribution of any living ursid . The species is distributed across much of northern Eurasia and North America . It is one of the two largest terrestrial carnivorans alive today , rivaled in body size only by its close cousin , the polar bear ( Ursus maritimus ) , which is much less variable in size and averages larger due to this . There are several recognized subspecies , many of which are quite well-known within their native ranges , found in the brown bear species . The brown bear 's principal range includes parts of Russia , Central Asia , China , Canada , the United States ( mostly Alaska ) , Scandinavia and the Carpathian region ( especially Romania ) , Anatolia , and Caucasus . The brown bear is recognized as a national and state animal in several European countries . While the brown bear 's range has shrunk and it has faced local extinctions , it remains listed as a least concern species by the International Union for Conservation of Nature ( IUCN ) with a total population of approximately 200,000 . As of 2012 , this and the American black bear are the only bear species not classified as threatened by the IUCN . However , the Californian , North African ( Atlas bear ) , and Mexican subspecies were hunted to extinction in the nineteenth and early twentieth centuries , and many of the southern Asian subspecies are highly endangered . One of the smaller-bodied subspecies , the Himalayan brown bear , is critically endangered , occupying only 2 % of its former range and threatened by uncontrolled poaching for its parts . The Marsican brown bear , one of several currently isolated populations of the main Eurasian brown bear race , in central Italy is believed to have a population of just 30 to 40 bears .", "ean sea level ( MSL ) ( abbreviated simply sea level ) is an average level of the surface of one or more of Earth 's oceans from which heights such as elevations may be measured . MSL is a type of vertical datuma standardised geodetic reference pointthat is used , for example , as a chart datum in cartography and marine navigation , or , in aviation , as the standard sea level at which atmospheric pressure is measured in order to calibrate altitude and , consequently , aircraft flight levels . A common and relatively straightforward mean sea-level standard is the midpoint between a mean low and mean high tide at a particular location . Sea levels can be affected by many factors and are known to have varied greatly over geological time scales . The careful measurement of variations in MSL can offer insights into ongoing climate change , and sea level rise has been widely quoted as evidence of ongoing global warming . The term above sea level generally refers to above mean sea level ( AMSL ) ." ] embeddings = model.encode(queries + passages) scores = model.similarity(embeddings[:2], embeddings[2:]) * 100 print(scores.tolist()) # [[66.12603759765625, 43.760101318359375], [47.67058563232422, 63.7889518737793]] ``` ### Transformers ``` python import torch import torch.nn.functional as F from torch import Tensor from transformers import AutoTokenizer, AutoModel def last_token_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor: left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0]) if left_padding: return last_hidden_states[:, -1] else: sequence_lengths = attention_mask.sum(dim=1) - 1 batch_size = last_hidden_states.shape[0] return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths] def get_detailed_instruct(task_description: str, query: str) -> str: return f'Instruct: {task_description}\nQuery: {query}' task = "Given a claim about climate change, retrieve documents that support or refute the claim" queries = [ get_detailed_instruct(task, "In Alaska, brown bears are changing their feeding habits to eat elderberries that ripen earlier."), get_detailed_instruct(task, "Local and regional sea levels continue to exhibit typical natural variability—in some places rising and in others falling.") ] passages = [ "The brown bear ( Ursus arctos ) is a large bear with the widest distribution of any living ursid . The species is distributed across much of northern Eurasia and North America . It is one of the two largest terrestrial carnivorans alive today , rivaled in body size only by its close cousin , the polar bear ( Ursus maritimus ) , which is much less variable in size and averages larger due to this . There are several recognized subspecies , many of which are quite well-known within their native ranges , found in the brown bear species . The brown bear 's principal range includes parts of Russia , Central Asia , China , Canada , the United States ( mostly Alaska ) , Scandinavia and the Carpathian region ( especially Romania ) , Anatolia , and Caucasus . The brown bear is recognized as a national and state animal in several European countries . While the brown bear 's range has shrunk and it has faced local extinctions , it remains listed as a least concern species by the International Union for Conservation of Nature ( IUCN ) with a total population of approximately 200,000 . As of 2012 , this and the American black bear are the only bear species not classified as threatened by the IUCN . However , the Californian , North African ( Atlas bear ) , and Mexican subspecies were hunted to extinction in the nineteenth and early twentieth centuries , and many of the southern Asian subspecies are highly endangered . One of the smaller-bodied subspecies , the Himalayan brown bear , is critically endangered , occupying only 2 % of its former range and threatened by uncontrolled poaching for its parts . The Marsican brown bear , one of several currently isolated populations of the main Eurasian brown bear race , in central Italy is believed to have a population of just 30 to 40 bears .", "ean sea level ( MSL ) ( abbreviated simply sea level ) is an average level of the surface of one or more of Earth 's oceans from which heights such as elevations may be measured . MSL is a type of vertical datuma standardised geodetic reference pointthat is used , for example , as a chart datum in cartography and marine navigation , or , in aviation , as the standard sea level at which atmospheric pressure is measured in order to calibrate altitude and , consequently , aircraft flight levels . A common and relatively straightforward mean sea-level standard is the midpoint between a mean low and mean high tide at a particular location . Sea levels can be affected by many factors and are known to have varied greatly over geological time scales . The careful measurement of variations in MSL can offer insights into ongoing climate change , and sea level rise has been widely quoted as evidence of ongoing global warming . The term above sea level generally refers to above mean sea level ( AMSL ) ." ] # load model and tokenizer tokenizer = AutoTokenizer.from_pretrained("zeta-alpha-ai/Zeta-Alpha-E5-Mistral") model = AutoModel.from_pretrained("zeta-alpha-ai/Zeta-Alpha-E5-Mistral") # get the embeddings max_length = 4096 input_texts = queries + passages batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors="pt") outputs = model(**batch_dict) embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask']) # normalize embeddings embeddings = F.normalize(embeddings, p=2, dim=1) scores = (embeddings[:2] @ embeddings[2:].T) * 100 print(scores.tolist()) [[66.15530395507812, 43.65541458129883], [47.681705474853516, 63.67986297607422]] ``` ### Zeta Alpha Zeta Alpha is the premier Neural Discovery Platform for AI and more. We are an Amsterdam-based R&D and product lab with a passion for AI technology, with offices on the Science Park campus of the University of Amsterdam. and in San Francisco. The Zeta Alpha Research team: - Arthur Câmara - Dinos Papakostas - Mathias Parisot - Fernando Rejon Barrera - Jakub Zavrel