Lajavaness/CrossEncoder-camembert-large

Model

Cross-Encoder Model for sentence-similarity

This model was is an improvement over the dangvantuan/CrossEncoder-camembert-large offering greater robustness and better performance

Training Data

This model was trained on the STS benchmark dataset and has been combined with Augmented SBERT. The model benefits from Pair Sampling Strategies using two models: CrossEncoder-camembert-large and dangvantuan/sentence-camembert-large. The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.

Usage (Sentence-Transformers)

Using this model becomes easy when you have sentence-transformers installed:

pip install -U sentence-transformers

Then you can use the model like this:

from sentence_transformers import CrossEncoder
model = CrossEncoder('Lajavaness/CrossEncoder-camembert-large', max_length=512)
scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ])

Evaluation

The model can be evaluated as follows on the French test data of stsb.

from sentence_transformers.readers import InputExample
from sentence_transformers.cross_encoder.evaluation import CECorrelationEvaluator
from datasets import load_dataset
def convert_dataset(dataset):
    dataset_samples=[]
    for df in dataset:
        score = float(df['similarity_score'])/5.0  # Normalize score to range 0 ... 1
        inp_example = InputExample(texts=[df['sentence1'], 
                                    df['sentence2']], label=score)
        dataset_samples.append(inp_example)
    return dataset_samples

# Loading the dataset for evaluation
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
df_test = load_dataset("stsb_multi_mt", name="fr", split="test")

# Convert the dataset for evaluation

# For Dev set:
dev_samples = convert_dataset(df_dev)
val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev')
val_evaluator(model, output_path="./")

# For Test set, the Pearson and Spearman correlation are evaluated on many different benchmark datasets:

test_samples = convert_dataset(df_test)
test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test')
test_evaluator(models, output_path="./")

Test Result: The performance is measured using Pearson and Spearman correlation:

• On dev

Model	Pearson correlation	Spearman correlation	#params
Lajavaness/CrossEncoder-camembert-large	90.34	90.15	336M
dangvantuan/CrossEncoder-camembert-large	90.11	90.01	336M

• On test:

Pearson score

Model	STS-B	STS12-fr	STS13-fr	STS14-fr	STS15-fr	STS16-fr	SICK-fr
Lajavaness/CrossEncoder-camembert-large	88.63	90.76	88.24	90.22	92.23	82.31	84.61
dangvantuan/CrossEncoder-camembert-large	88.16	90.12	88.36	89.86	92.04	82.01	84.23

Spearman score

Model	STS-B	STS12-fr	STS13-fr	STS14-fr	STS15-fr	STS16-fr	SICK-fr
Lajavaness/CrossEncoder-camembert-large	88.03	84.87	87.88	89.10	92.16	82.50	80.78
dangvantuan/CrossEncoder-camembert-large	87.57	84.24	88.01	88.62	91.99	82.16	80.38