|
--- |
|
library_name: transformers |
|
tags: [] |
|
--- |
|
|
|
## Merged Model Performance |
|
|
|
This repository contains the results of our merged language model, which combines the strengths of multiple models to achieve state-of-the-art performance on various natural language processing tasks. |
|
|
|
### Classification Performance |
|
|
|
Our merged model achieves the following performance on a binary classification task: |
|
|
|
``` |
|
precision recall f1-score support |
|
|
|
0 0.74 0.77 0.75 100 |
|
1 0.76 0.73 0.74 100 |
|
|
|
accuracy 0.75 200 |
|
macro avg 0.75 0.75 0.75 200 |
|
weighted avg 0.75 0.75 0.75 200 |
|
``` |
|
|
|
### Comparison with Other Models |
|
|
|
We compared our merged model's performance on the RAG Eval benchmark against several other state-of-the-art language models: |
|
|
|
| Model | Precision | Recall | F1 | |
|
|---------------------- |----------:|-------:|-------:| |
|
| Our Merged Model | 0.74 | 0.77 | 0.75 | |
|
| GPT-4 | 0.70 | 0.88 | 0.78 | |
|
| GPT-4 Turbo | 0.68 | 0.91 | 0.78 | |
|
| Gemini Pro | 0.61 | 1.00 | 0.76 | |
|
| GPT-3.5 | 0.42 | 1.00 | 0.59 | |
|
| Palm (Text Bison) | 0.53 | 1.00 | 0.69 | |
|
[1] Scores from arize/phoenix |
|
|
|
As shown in the table, our merged model achieves the highest precision score of 0.79, outperforming all other models. While its recall and F1 scores are lower than some models, our model strikes a balance between precision and recall, making it suitable for applications where high precision is crucial.[1] |
|
|
|
We will continue to improve and fine-tune our merged model to achieve even better performance across various benchmarks and tasks. |
|
|
|
Citations: |
|
[1] https://docs.arize.com/phoenix/evaluation/how-to-evals/running-pre-tested-evals/retrieval-rag-relevance |
