Adding the Open Portuguese LLM Leaderboard Evaluation Results
#4
by
leaderboard-pt-pr-bot
- opened
README.md
CHANGED
@@ -1,5 +1,8 @@
|
|
1 |
---
|
2 |
-
|
|
|
|
|
|
|
3 |
tags:
|
4 |
- mistral
|
5 |
- instruct
|
@@ -8,13 +11,10 @@ tags:
|
|
8 |
- gpt4
|
9 |
- synthetic data
|
10 |
- distillation
|
|
|
11 |
model-index:
|
12 |
- name: OpenHermes-2-Mistral-7B-PTBR
|
13 |
results: []
|
14 |
-
license: apache-2.0
|
15 |
-
language:
|
16 |
-
- en
|
17 |
-
- pt
|
18 |
---
|
19 |
|
20 |
# OpenHermes 2 - Mistral 7B - PT BR
|
@@ -103,4 +103,23 @@ To utilize the prompt format without a system prompt, simply leave the line out.
|
|
103 |
Currently, I recommend using LM Studio for chatting with Hermes 2. It is a GUI application that utilizes GGUF models with a llama.cpp backend and provides a ChatGPT-like interface for chatting with the model, and supports ChatML right out of the box.
|
104 |
In LM-Studio, simply select the ChatML Prefix on the settings side pane:
|
105 |
|
106 |
-
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/ls6WqV-GSxMw2RA3GuQiN.png)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
language:
|
3 |
+
- en
|
4 |
+
- pt
|
5 |
+
license: apache-2.0
|
6 |
tags:
|
7 |
- mistral
|
8 |
- instruct
|
|
|
11 |
- gpt4
|
12 |
- synthetic data
|
13 |
- distillation
|
14 |
+
base_model: mistralai/Mistral-7B-v0.1
|
15 |
model-index:
|
16 |
- name: OpenHermes-2-Mistral-7B-PTBR
|
17 |
results: []
|
|
|
|
|
|
|
|
|
18 |
---
|
19 |
|
20 |
# OpenHermes 2 - Mistral 7B - PT BR
|
|
|
103 |
Currently, I recommend using LM Studio for chatting with Hermes 2. It is a GUI application that utilizes GGUF models with a llama.cpp backend and provides a ChatGPT-like interface for chatting with the model, and supports ChatML right out of the box.
|
104 |
In LM-Studio, simply select the ChatML Prefix on the settings side pane:
|
105 |
|
106 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/ls6WqV-GSxMw2RA3GuQiN.png)
|
107 |
+
|
108 |
+
|
109 |
+
# Open Portuguese LLM Leaderboard Evaluation Results
|
110 |
+
|
111 |
+
Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/artificialguybr/OpenHermesV2-PTBR) and on the [π Open Portuguese LLM Leaderboard](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard)
|
112 |
+
|
113 |
+
| Metric | Value |
|
114 |
+
|--------------------------|---------|
|
115 |
+
|Average |**61.94**|
|
116 |
+
|ENEM Challenge (No Images)| 59.90|
|
117 |
+
|BLUEX (No Images) | 48.40|
|
118 |
+
|OAB Exams | 41.41|
|
119 |
+
|Assin2 RTE | 88.89|
|
120 |
+
|Assin2 STS | 67.61|
|
121 |
+
|FaQuAD NLI | 53.06|
|
122 |
+
|HateBR Binary | 74.77|
|
123 |
+
|PT Hate Speech Binary | 64.38|
|
124 |
+
|tweetSentBR | 59.01|
|
125 |
+
|