metadata

base_model: microsoft/mpnet-base
datasets:
  - mteb/askubuntudupquestions-reranking
  - mteb/mind_small
  - mteb/scidocs-reranking
  - mteb/stackoverflowdupquestions-reranking
  - mteb/arguana
  - mteb/climate-fever
  - mteb/cqadupstack-android
  - mteb/cqadupstack-english
  - mteb/biosses-sts
  - mteb/sickr-sts
  - mteb/sts12-sts
  - mteb/sts13-sts
language:
  - en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:5130135
  - loss:MultipleNegativesSymmetricRankingLoss
  - loss:CoSENTLoss
  - dataset_size:8233
widget:
  - source_sentence: This is a sample source sentence.
    target_sentence: This is a sample target sentence.
    sentences:
      - >-
        Broadband is a necessary evolution of internet technology that firms
        would be wise to avail of if they wish to remain competitive. But it is
        this very desirability that makes the provision of broadband a lucrative
        business in which many firms participate. Business on a large scale is
        rarely organised in diffuse patterns, but clustered in major population
        centres. Economic development can be furnished by the private sector
        investing in broadband where there is a market. Growth will not be
        slowed just because some farmers in Nebraska have slower internet.
        Singapore is an aberrant example, as it is so small and its population
        so dense that it would be impossible to compare its provision of
        broadband access to most other countries.
      - >-
        In retrospect, the decision to welcome the former Soviet states in the
        Baltic into NATO appears foolish. They continue to have a prickly
        relationship with Russia, which has some legitimate concerns about the
        treatment of large Russian minorities in Latvia and Estonia, and about
        the siting of US nuclear defences. Their entry into NATO was forced upon
        Russia, which naturally saw it as an aggressive move designed to
        humiliate it, and marked the point when its pro-western policy shifted
        to a more nationalist and confrontational approach. [1] It also weakened
        the unity of NATO as there are quite legitimate doubts as to whether all
        the alliance’s members would really go to war with Russia over the
        integrity of, say, Estonia. Given this history, it would be madness to
        compound the problem by extending NATO membership to Georgia and
        Ukraine.  [1] Fraser, Malcolm, ‘Ukraine: there’s no way out unless the
        west understands its past mistakes’, theguardian.com, 3 March 2014,
      - >-
        We need to be critical of the cumulative potential of the tax model
        proposed. Firstly, the theory of the state’s capacity and how it
        functions in practice differ substantially. The idea of taxation acting
        to enhance the productive capacity of a nation is based on assumptions
        that the institutions, human resources, and state-capacity, are already
        present. This is not always the case in Africa.  Corruption and bad
        governance are prevalent. Reforms in 1996 to curb corruption in the TRA
        were reversed due to misunderstanding the nature of corruption amongst
        tax officials and administration (Fjelstad, 2003). Tax-revenue
        performance remains comparatively low [1] , there is little reason to
        simply altering what taxes there are will change this.  Finally,
        alternative methods can be used to assist rural infrastructure projects,
        and enable national savings. For example, revising the role of
        agricultural marketing boards [2] .  [1] See further readings: Gray and
        Kahn, 2010.  [2] See further readings: Baffes, 2005.
  - source_sentence: advantages/disadvantages of installing from source code
    sentences:
      - >-
        is it there any advantage to unticking the `` source code '' entry of a
        repository ? what is the advantage to build unity from source ffmpeg :
        installing from repository or compiling from source ?
      - >-
        flash player issues flash player alternative for firefox flash videos go
        fullscreen once ( firefox ) , then after minimising it and trying to go
        fullscreen again it freezes ( ubuntu 14.04 , gnome desktop ) choppy
        flash video playback , 'settings ' disabled for flash as well flash
        player is not working and do n't know what to do to fix it flash player
        : sound distorted error flash chrome in ubuntu 14.04
      - >-
        how to install newest version of minitube from source installing two
        version of a software how to find source for a line in $ path ? how do i
        install network driver source code to /lib/modules/ < kernel version >
        /build/drivers/net/ethernet ca n't install vmware-view-client from
        partner repo in saucy why apt-get does n't install the newest software
        what is the right way to reinstall from source after package was
        installed via apt ? do i have to remove open drivers to switch to fglrx
        ? how do i install clang version 3 in 10.10 ? adding extract tool to
        nautilus ( compiled from source code ) how to point to boost built from
        source how to remove httpd built from source if i build a package from
        source how can i uninstall or remove completely ? avoid reinstall
        virtualbox and update to the latest release installing only ruby 1.9 can
        i update software installed from source code directly determine which
        package ( s ) were installed from a particular source
  - source_sentence: will i be able to upgrade to 13.04 from 12.10 with a wubi installation ?
    sentences:
      - add cron job on startup from a script
      - >-
        upgrading from 12.10 wubi install to 13.04 can i safely upgrade ubuntu
        12.10 to 13.04 which installed using wubi ? will i be able to use wubi
        to install ubuntu releases newer than 12.10 ? installing ubuntu in
        windows 8 with wubi
      - >-
        upgrade from ubuntu server 12.10 to 13.04 wubi after upgrading windows 7
        to windows 8 should i use wubi ubuntu 12.04 for my academic studies like
        writing docs , programming , etc ? how to do a fresh re-installation of
        ubuntu safely on dual boot ? wubi 12.10 installation on windows 8 hangs
        do i keep my windows installation if i install ubuntu with wubi , the
        windows installer ? move wubi installation of ubuntu to a different
        partition in windows failed why was wubi removed from 13.04+ backup ,
        install and restore programms/settings of ubuntu ( wubi ) installation
        unable to install ubuntu 12.10 with wubi in windows 7 - wubi is stuck
        upgrading my wubi 11.04 to 11.10 no additional driver no usb or dvd
        drive and not able to migrate wubi ( no clue ) how can i remove windows
        and upgrade wubi install to a full install without a cd or usb stick ?
        upgrade ubuntu to 13.04 from 12.10 on dual-boot laptop upgrading ubuntu
        12.10 to 13.04 using bootable usb alongside windows 7
  - source_sentence: how do i use ubuntu 's web application integration ?
    sentences:
      - >-
        out of the box ubuntu 13.04 with chromium does not automatically use web
        apps
      - >-
        why i do not have gnome shell with gnome 3 ? no gnome shell after
        install on ubuntu 12.04 inside vmware workstation
      - >-
        how do i put a web application on the launcher ? how can i force ubuntu
        to use font a instead font b in all applications and web pages ? how to
        remove the web app shortcut created by chromium ? is there any finished
        tutorial on how to develop and package unity web apps ? how do i put web
        applications in my unity launcher ? how to ftp transfer files to
        /var/www ? i would like to set up a ubuntu os on amazon web server , how
        do i do this is there a way to request support for web applications not
        currently supported ? how to test web applications for mobile devices on
        ubuntu ? can i use unity web apps in ubuntu 13.10 ? how to create
        chrome/chromium web application desktop shortcut ? icedtea-web 1.5 needs
        confirmation to run trusted applet xul + unity web api hp pavilion
        dv2000 web camera driver how to open applications after booting a purely
        command line interface ? trouble connecting to wireless via clear hub
        express are web apps still part of ubuntu 's future ? newer versions of
        chromium ? powerpc macs and 12.04 lts application support
  - source_sentence: >-
      This form of necrosis, also termed necroptosis, requires the activity of
      receptor-interacting protein kinase 1 (RIP1) and its related kinase, RIP3 
    sentences:
      - >-
        This large-scale study showed that IDH1/IDH2 mutations were mutually
        exclusive with inactivating TET2 mutations, suggesting that the two
        types of mutations had similar effects and were thus functionally
        redundant.
      - >-
        Co-transfection of miRVec-miR-204 and the Renilla-3′ UTR plasmid was in
        HEK293T cells with TransIT-LT1 Transfection Reagent (Mirus)
      - >-
        TNF-mediated programmed necrosis typically involves the
        receptor-interacting serine-threonine kinases 1 and 3 (RIP1 and RIP3),
        as evidenced in human, mouse, and zebrafish cell lines, as well as in a
        murine sepsis model

SentenceTransformer based on microsoft/mpnet-base

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the reranking_1, retrival_1 and sts_1 datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: microsoft/mpnet-base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity
Training Datasets:
Language: en

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'This form of necrosis, also termed necroptosis, requires the activity of receptor-interacting protein kinase 1 (RIP1) and its related kinase, RIP3 ',
    'TNF-mediated programmed necrosis typically involves the receptor-interacting serine-threonine kinases 1 and 3 (RIP1 and RIP3), as evidenced in human, mouse, and zebrafish cell lines, as well as in a murine sepsis model',
    'This large-scale study showed that IDH1/IDH2 mutations were mutually exclusive with inactivating TET2 mutations, suggesting that the two types of mutations had similar effects and were thus functionally redundant.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Datasets

reranking_1

Dataset: reranking_1 at 4d853f9
Size: 337 training samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 4 tokens mean: 14.94 tokens max: 39 tokens	min: 5 tokens mean: 84.1 tokens max: 407 tokens	min: 2 tokens mean: 188.2 tokens max: 391 tokens

Samples:

anchor	positive	negative
`black screen after installation with wubi`	will installing ubuntu harm my computer ? wubi boot problem ca n't install on an emachines e725 black screen after purple ubuntu screen error , then black screen after wubi 11.10 goes into purple screen with black squares trying to login ( using wubi ) system hangs after wubi install screen turns off on boot dark purple screen with graphic corruption at boot , using wubi can i install 32 bit ubuntu on dual boot with 64 bit windows 7 ? black screen during installation ubuntu 12.10 via wubi on hp pavillion g6-1273s1 ubuntu wont boot after installing wubi from windows 7 windows 8 dual boot problem dual boot , ubuntu wo n't boot ! second wubi installation not working ubuntu after boot shows a black screen with a warning reboot required	`how to fix black screen after boot purple then black screen after booting ? black screen after the grub screen ca n't boot after update , purple/black screen`
`wubi gives error 'nonetype ' object has no attribute 'get_info '`	nonetype object has no attribute get_info ubuntu 14.04 can not install ubuntu 13.04 using wubi why does wubi installation fail with `` error : 'none type ' object has no attribute 'get_info ' '' ? error - 'nonetype ' object has no attribute 'get_info ' during xubuntu installation ca n't install 12.04 inside windows 7 using wubi error while installing ubuntu as separate os on my pc ? wubi 's error while installin ubuntu 12.10 on windows 7 why is wubi giving me an error message when i try to install ? ubuntu 12.10 install fails , both native and wubi asus eeepc 1015cx ubuntu 32 bits or 64bits ?	attributeerror : 'module ' object has no attribute openshot does n't launch with failed to import '' message software center not working ubuntu one not syncing ... in windows xp sp3 ubuntu one for windows client : attributeerror '' messages ubuntu one sign in error ( after installation completed ) : nonetype '' object has no attribute make file '' `` attributeerror : 'nonetype ' object has no attribute 'group ' '' when trying to use youtube-dl ' '' windows backend object has no attribute 'iso-path ' - see log for details . ' error when trying to install how can i resolve a 'windows backend : object has no attribute iso_path ' error while installing with wubi ? 'module ' object has no attribute 'py2 '
`give permission to /var/www`	write permissions in /var/www folder folders and files permission problem user ca n't ftp to a directory but is part of the group that owns it how to give read write permission to a folder and its sub folders and files ? give apache permission to write to /home/*/www/ directories give www-data write permission to /home ? assining permission to www folder problems with the /var/www folder trouble accessing www folders - permission or ownership ? group access to directory i am part of a group , but i can not create files apache 'you do n't have permission to access / on this server '	`missing /var/www lubuntu 13.04 change permission from -rwxrwxr-x to drwxrwxr-x wordpress upload permission on nginx & ubuntu how do i restore the default permission on a directory/folder ? file permission in www/html directory for security problem while installing visual paradigm uml tool ? setting up home folders for users for 12.04 server edition -ls : can not open directory . : permission denied changed apache www folder`

Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

retrival_1

Dataset: retrival_1 at c22ab2a
Size: 7,806 training samples
Columns: title and text
Approximate statistics based on the first 1000 samples:
title text
type string string
details
min: 2 tokens
mean: 5.42 tokens
max: 17 tokens

min: 41 tokens
mean: 199.3 tokens
max: 512 tokens

	title	text
type	string	string
details	min: 2 tokens mean: 5.42 tokens max: 17 tokens	min: 41 tokens mean: 199.3 tokens max: 512 tokens

Samples:

title	text
	Not having children is environmentally friendly The more people consume in the world, the greater the environmental damage. An average American produces 52 tons of garbage by the age of 75.* However, producing extra litter and pollution is not the only hazard that every child poses to the planet. Increasing world’s population also places incredible stress on Earth’s resources. It is estimated, for instance, that by 2025 three billion people will live in water-scarce countries. By reducing the number of human beings we will manage to avoid numerous overpopulation crises and reverse the damage done to the environment. * Tufts Climate Initiative., 2006,
	`The need for interaction is all the more reason to ensure that all ideas are in the marketplace. This way, the veracity of all ideas are questioned. For example, if someone brings bigoted ideas with them as a freshman, perhaps because these ideas were prevalent in the community they grew up in, if they cannot express these ideas and be challenged they may never attempt to integrate. Instead, they will gravitate to those who share their ideas and remain isolated.`
	Putting the power to censor the internet, no matter how stringent or specific the guidelines, into the hands of a private organization is misguided. It is the state not individual ISPs who are needed to assess how dangerous a site is, whether it is actually promoting extremism, and ultimately make a decision as to whether a site needs to be blocked. The ISPs may end up being the actors that implement the policy but it has to be government that decides which websites to block and why. This also means that the decision would be much more centralised. Leaving this decision to the discretion of individual ISPs will mean that some websites will be blocked on some ISPs and not on others. Only government can ensure that there is consistency.

Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

sts_1

Dataset: sts_1 at 9ee918f
Size: 90 training samples
Columns: sentence1, sentence2, and score
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 score
type string string float
details
min: 13 tokens
mean: 39.44 tokens
max: 88 tokens

min: 12 tokens
mean: 38.43 tokens
max: 95 tokens

min: 0.0
mean: 2.24
max: 4.0

	sentence1	sentence2	score
type	string	string	float
details	min: 13 tokens mean: 39.44 tokens max: 88 tokens	min: 12 tokens mean: 38.43 tokens max: 95 tokens	min: 0.0 mean: 2.24 max: 4.0

Samples:

sentence1	sentence2	score
`As p16INK4a blocks the inactivation of pRb by cyclin-dependent kinases, and Arf blocks the inactivation of p53 by Mdm2-mediated proteolysis, both have the capacity to cause cell cycle arrest.`	`By inducing Arf and Ink4a in primary rodent fibroblasts, oncogenic Ras expression leads to growth arrest and premature senescence.`	`2.4`
`Several lines of evidence suggest that the known principal RB pathway lesions in human tumors act in a mutual exclusive manner.`	`In individual human tumor specimens, these principal components of the pathway—RB-CDK4/6-p16INK4A—are reported to be targeted in a mutually exclusive manner.`	`3.4`
`T47D, MCF-7, Skbr3, HeLa, and Caco-2 cells were transfected by electroporation as described previously.`	`MCF7 or HeLa cells were electroporated as described previously to more than 95% efficiency with pSuper constructs against the various targets, and 72 hr later, protein expression was analyzed by SDS-PAGE and Western blotting.`	`3.0`

Loss: CoSENTLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Evaluation Datasets

reranking_1

Dataset: reranking_1 at 4d853f9
Size: 38 evaluation samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 7 tokens mean: 14.0 tokens max: 24 tokens	min: 10 tokens mean: 77.58 tokens max: 371 tokens	min: 2 tokens mean: 196.55 tokens max: 291 tokens

Samples:

anchor	positive	negative
how do i clean install windows 7 on the `` other '' partition ?	`how to partition disk in ubuntu and then install windows 7 ? install windows 7 on seperate partitions without losing grub how do i create a partion to install windows ? doubt : about dual boot ( 14.04 with windows 7 ) how do i install windows 7 from an iso file on a separate partition from ubuntu ? how to partition for a dual boot`	install ubuntu in a specific partition side by side with windows 7 windows 7 system partition i have xp on one partition and windows 7 on another . how do i install ubuntu on a seperate partition ? cant ' dual install xubuntu and windows 7 i have formatted a partition for ubuntu . how do i install it on that partition ? grub wo n't load windows 7 ( unknown file system ) how to delete windows xp and integrate its partition to ubuntu ? uninstalling ubuntu when it is on the windows partition upgrading current version of ubuntu , on a multi boot system . a partition is necessary before installing ubuntu on windows 7 ? how to do a clean reinstall of ubuntu ? can i use testdisk to recover the windows partition over which ubuntu was installed ? how to update dual boot windows 7 / ubuntu 13.04 to 14.04 if i chose `` replace windows with ubuntu '' , do i lose the other partition too ?
`is it possible to have two different dpi configurations for two different screens ?`	`how can i change the dpi of my monitors separately ? using external monitor with laptop monitor as separate monitors resolution in login screen different text scaling factors for different displays ? external display is showing and reporting wrong resolutions hard drive encryption boot screen resolution how to disable clone display mode on ubuntu login screen ?`	change logon screen display configuration login screen on the wrong monitor only one resolution available in xorg.conf force gdm login screen to the primary monitor how to find and change the screen dpi ? dual screen different font resolution how can i display laptop screen on the tv by hdmi in kubuntu how do i turn off the laptop screen while using an external monitor ? adapt ubuntu to a high-dpi resolution screen how can i change my login screen resolution when connected to my monitor ( 12.04 ) dpi setting in ubuntu 13.10 resolution of the login screen is wrong on a second monitor force full-screen game to one monitor ?
`any command line calculator for ubuntu ?`	`quick simple mathematics calculations ubuntu calculator in dash`	shortcut to open/focus terminal window how to assign shortcut keys in gnome ? menu bar not visible on gnome-calculator how to permanently enable in-dash calculator in 13.10 a good hex/binary calculator command line calculator that keeps fractional values is there any calculator in software centre which could solve quadratic equations ? how to make shortcut to terminal application ? can calculator show decimal as fraction ? how to open pseudo-terminal devices on terminal ? keep calculator from opening a new window every time i press the `` calculator '' button on the keyboard ? keyboard shortcut for terminal does n't work since updating to 12.10 decimal point from number pad wo n't work on the calculator but will everywhere else how to set qalculator as default calculator ? what is 'calculator ' package ? how to get a 'non-screenlet ' calculator in compiz widget layer ? how do i use the 'scientific ' mode in the calculator ? how can i capture text from my terminal without redirecting it to a text file ?

Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

retrival_1

Dataset: retrival_1 at c22ab2a
Size: 868 evaluation samples
Columns: title and text
Approximate statistics based on the first 1000 samples:
title text
type string string
details
min: 2 tokens
mean: 5.0 tokens
max: 17 tokens

min: 26 tokens
mean: 204.03 tokens
max: 512 tokens

	title	text
type	string	string
details	min: 2 tokens mean: 5.0 tokens max: 17 tokens	min: 26 tokens mean: 204.03 tokens max: 512 tokens

Samples:

title	text
	Behind the veil of ignorance, human beings may not in fact side with what gives them the statistical greatest chance of survival. As Rawls himself notes, people are naturally risk-averse, and thus will select the rules that protect them from the worst possible situations, even if that sacrifice would help many others. Most people find the prospect of being actively killed by the conscious action of another human being worse than simply dying in an accident, and would seek to protect themselves against that worse outcome.
	Schengen membership is not the same as EU membership – some non-EU states, such as Switzerland are part of Schengen, the UK and Ireland are EU member states but are not. Joining Schengen would involve the politically sensitive issue of undocumented migrants, which could not only be fatal to Cape Verde joining Schengen but to integration with Europe itself. Even if it is unlikely, is it that difficult for people to show a passport? Besides, tourism is not just from Europe to outside – a Euro move would only stop Europeans from needing to change currencies. The peg is the best of both worlds in that it means that the currency is stable.
	News organisations cannot be completely transparent if they are to do their job properly and News International is no exception. Such organisations cannot for example reveal their sources as this may sometimes put their sources at risk and would mean that others would not come forward. As part of this news companies need to keep secret how they obtained information. While an attempt by a newspaper to cover up crimes is regrettable this one newspapers actions should not tar the whole company and its other papers.

Loss: MultipleNegativesSymmetricRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

sts_1

Dataset: sts_1 at 9ee918f
Size: 10 evaluation samples
Columns: sentence1, sentence2, and score
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 score
type string string float
details
min: 17 tokens
mean: 38.4 tokens
max: 61 tokens

min: 27 tokens
mean: 46.8 tokens
max: 62 tokens

min: 0.0
mean: 1.76
max: 3.2

	sentence1	sentence2	score
type	string	string	float
details	min: 17 tokens mean: 38.4 tokens max: 61 tokens	min: 27 tokens mean: 46.8 tokens max: 62 tokens	min: 0.0 mean: 1.76 max: 3.2

Samples:

sentence1	sentence2	score
`Recently, it was reported that expression of IDH1R132H suppresses TET2 activity and the mutations of IDH1 and IDH2 genes occur in a mutual exclusive manner with that of TET2 gene in AML`	`The mechanism was clarified by yet another genomic survey, this time involving acute myelogenous leukemia (AML).`	`1.4`
`Recently, it was reported that expression of IDH1R132H suppresses TET2 activity and the mutations of IDH1 and IDH2 genes occur in a mutual exclusive manner with that of TET2 gene in AML.`	`This large-scale study showed that IDH1/IDH2 mutations were mutually exclusive with inactivating TET2 mutations, suggesting that the two types of mutations had similar effects and were thus functionally redundant.`	`3.2`
`A gene that warrants further studies is the erythropoietin receptor that is 7.4-fold higher expressed in TEL-AML1-positive cases compared to other precursor B-ALL cases confirming other gene expression classification studies.`	`Another recent gene expression study of large numbers of cases provided support for the hypothesis that distinct leukemias are specified by each of the unique chromosomal abnormalities found in lymphoblastic leukemias.`	`0.6`

Loss: CoSENTLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 2
per_device_eval_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 2e-05
num_train_epochs: 5
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: True
tf32: True
optim: adamw_torch_fused
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 2
per_device_eval_batch_size: 2
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 8
eval_accumulation_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 5
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	reranking 1 loss	retrival 1 loss	sts 1 loss
0.9716	500	0.4296	-	-	-
0.9988	514	-	0.2080	0.1297	0.1096
0.9716	500	0.0933	-	-	-
0.9988	514	-	0.2104	0.1012	0.5515
0.9716	500	0.0435	-	-	-
0.9988	514	-	0.2026	0.1434	0.4824
0.9716	500	0.0282	-	-	-
0.9988	514	-	0.1827	0.1305	0.2950
1.9432	1000	0.0555	-	-	-
1.9995	1029	-	0.3932	0.0693	0.8649
2.9147	1500	0.0151	-	-	-
2.9983	1543	-	0.2112	0.0555	0.5478
3.8863	2000	0.0036	-	-	-
3.9990	2058	-	0.1921	0.0432	0.5912
4.8579	2500	0.0013	-	-	-
4.9939	2570	-	0.1904	0.0412	0.6356

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.0.1
Transformers: 4.41.2
PyTorch: 2.2.0+cu121
Accelerate: 0.31.0
Datasets: 2.20.0
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}