File size: 24,653 Bytes
e2411e7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 |
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: "Evaluation:\n\n1. **Context Grounding**: The answer references the document,\
\ but it consists of multiple points that are not directly related to accessing\
\ training resources. While some methods mentioned might facilitate training (e.g.,\
\ learning budget), others (like using 1Password, Tresorit) are not directly relevant\
\ to accessing training resources.\n \n2. **Relevance**: The answer partially\
\ addresses the question. Points about accessing documents or requesting a learning\
\ budget are somewhat related, but the inclusion of security tools and password\
\ managers is irrelevant to the question of accessing training resources.\n\n\
3. **Conciseness**: The answer includes unnecessary details that do not directly\
\ answer the question, making it lengthy and off-point in segments.\n\n4. **Specificity**:\
\ The answer is a mix of specific steps and unrelated information. It fails to\
\ provide a direct method to access training resources comprehensively.\n\n5.\
\ **General Tips**: The answer does provide a step about talking to a lead for\
\ a learning budget, which can be considered a relevant tip. However, it's buried\
\ in a lot of unrelated content.\n\nOverall, the answer deviates too much from\
\ the specific question about accessing training resources and includes information\
\ not requested nor directly relevant.\n\nFinal result: **Bad**"
- text: 'Evaluation:
The answer provided is concise and directly addresses the question of whom to
contact for travel reimbursement questions. It correctly refers to the email address
provided in the document. The answer is well-supported by the document and does
not deviate into unrelated topics.
The final evaluation: Good'
- text: 'Reasoning:
1. **Context Grounding**: The answer accurately references content from the provided
documents, especially the points related to actively thinking about the possibility
of someone leaving, flagging it to HR, analyzing problems, and providing feedback.
2. **Relevance**: The answer directly addresses the question by outlining why
it is crucial for team leads to consider the possibility of staff leaving and
the steps they can take to mitigate issues early on.
3. **Conciseness**: The answer is relatively concise but gets slightly verbose
towards the end. The core points are clearly made without too much unnecessary
information.
4. **Specificity**: The answer includes specific reasons like addressing underperformance,
lack of growth, disagreement with direction, and maintaining a supportive environment,
all of which are well-supported by the documents.
5. **Avoiding Generality**: The answer provides detailed steps and reasons as
mentioned in the documents, avoiding overly general statements.
Final Result: **Good**'
- text: 'Evaluation:
The answer addresses the question partially by suggesting ways to learn about
ORGANIZATION through their website and available job postings, which are referenced
in the documents. However, it misses more specific ways to understand their products
and challenges; for example, details from the document about stress management,
inclusivity issues, and organizational changes could provide better insights into
their future and challenges. Additionally, a newsletter about "behind the scenes"
is more about updates rather than specific details on products and challenges.
Therefore, the answer lacks completeness and specificity.
The final evaluation: Bad'
- text: "**Evaluation Reasoning:**\n\n1. **Context Grounding:** The answer is well-supported\
\ by the provided documents, especially Document 1. It correctly identifies the\
\ responsibilities of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera.\n\
\ \n2. **Relevance:** The answer addresses the specific question about the role\
\ of ORGANIZATION_2 in the farewell process.\n \n3. **Conciseness:** The response\
\ includes some redundant information, especially the repetition of individuals\
\ involved and some operational details not directly related to the extent of\
\ ORGANIZATION_2's involvement.\n \n4. **Specificity:** It clearly outlines\
\ the roles and describes the process for different situations, including handling\
\ paperwork, process, and tough conversations.\n \n5. **General vs. Specific\
\ Tips:** The answer could have been more concise regarding the involvement extent\
\ but remains within bounds.\n\n**Final Evaluation: Good**"
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
results:
- task:
type: text-classification
name: Text Classification
dataset:
name: Unknown
type: unknown
split: test
metrics:
- type: accuracy
value: 0.6119402985074627
name: Accuracy
---
# SetFit with BAAI/bge-base-en-v1.5
This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
The model has been trained using an efficient few-shot learning technique that involves:
1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.
## Model Details
### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->
### Model Sources
- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
### Model Labels
| Label | Examples |
|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | <ul><li>"Evaluation:\nThe answer is too general and lacks specificity regarding the organization's guidelines and criteria for spending money wisely. The provided documents are focused on the study and learning budget within the organization, and the answer should have included aspects such as considering whether an expenditure is a good investment, its alignment with personal and organizational goals, and the outlined budget guidelines.\n\n1. **Context Grounding:** The answer does touch upon wise spending and investments, but it lacks direct references to the specific criteria and guidelines outlined in the documents.\n2. **Relevance:** The answer is somewhat relevant but doesn't delve into the detailed criteria provided about spending the budget wisely.\n3. **Conciseness:** The answer is concise but misses key specifics from the document.\n4. **Avoiding unanswerable responses:** The question could be answered based on the document, and the answer does attempt to respond appropriately.\n5. **Specificity and details:** The answer misses details such as the control questions about personal and organizational benefits, the role and responsibility match, and the specific budget thresholds for various spending categories.\n\nFinal evaluation: Bad"</li><li>'Evaluation:\n\n1. **Context Grounding:** The answer is somewhat grounded in the context provided by the documents but not entirely accurate as it contains errors such as repeated names and unclear sentences.\n2. **Relevance:** The answer does address the question by listing pet peeves, yet it includes unnecessary names and errors that detract from its clarity.\n3. **Conciseness:** The response is not concise and includes redundant information, such as repeated names and irrelevant phrases.\n4. **Clarity:** The answer is marred by text errors (e.g., "Cassandra Rivera Heather Nelson"), which obscure its clarity and readability.\n5. **Specificity:** It does list specific pet peeves mentioned in the document but lacks proper structure and clarity.\n\nFinal evaluation: Bad'</li><li>'The provided answer is inadequate for several reasons:\n\n1. **Context Grounding**: The answer makes vague references to multiple unrelated sections, rather than directly addressing how to access training resources. The mentioned documents do not collectively substantiate the response about accessing training resources.\n\n2. **Relevance**: The instructions involve details about reimbursement, security practices, and learning budgets, but they do not respond directly to the query about accessing training resources.\n\n3. **Conciseness**: The answer includes numerous irrelevant points such as the use of 1Password for personal account management and feedback processing, which are not pertinent to accessing training resources.\n\n4. **Specificity**: The answer does not provide specific steps or tips directly related to accessing the training resources of the company. It rather diverts into various other topics.\n\n5. **Following Instructions**: The given documents offer information about other aspects of work at the company but lack direct guidance on accessing training resources. Thus, attempting to answer the question without appropriate grounding in the document misleads the response.\n\nTherefore, the answer fails to meet the necessary criteria and should be evaluated as:\n\n**Bad**'</li></ul> |
| 1 | <ul><li>'Evaluation:\nThe provided answer addresses the specific question of how feedback should be given according to the provided tips. It covers key points from the document such as giving feedback at the time of the event, making it situational rather than personal, reflecting on the intention of the feedback, being clear and direct, and showing appreciation. Additionally, it touches on tips for receiving feedback, aligning well with the original document.\n\nHowever, the answer includes erroneous and irrelevant information due to text misanalyzation such as "Christopher Estes time" which should not be part of the answer. Furthermore, the mention of "emichelle James Johnson MDamples" appears to be a corrupted text which doesn’t make sense in the context.\n\nFinal evaluation: Bad'</li><li>'Evaluation:\nThe answer accurately addresses the question of why it is important to proactively share information from high-level meetings. It discusses transparency, alignment, sense of purpose, and open communication, which are all points grounded in the provided documents. The answer is relevant, concise, and specific enough, capturing the essence of the reasons mentioned in Document 4. No irrelevant information is included, and the answer contains specific tips related to the importance of sharing high-level meeting details.\n\nFinal evaluation: Good'</li><li>'Evaluation:\nThe provided answer is generally well-grounded in the document, sticking closely to the instructions given for reporting car travel expenses for reimbursement. However, there are a few inaccuracies and incomplete information:\n\n1. **Context Grounding**: The answer is largely based on the content of the document but not entirely accurate. The provided email addresses seem to be incorrectly formatted (e.g., "finance@Dustin [email protected]" should probably be "[email protected]@example.net").\n2. **Relevance**: The response is relevant to the question asked and sticks to the specific procedure for car travel expense reporting.\n3. **Conciseness**: The answer is concise but could be more accurate in presenting the information.\n4. **Specificity**: The document also mentions tracking kilometers and the reimbursement rate, which the answer includes. However, it fails to mention all the specific details accurately.\n\nFinal evaluation: Bad'</li></ul> |
## Evaluation
### Metrics
| Label | Accuracy |
|:--------|:---------|
| **all** | 0.6119 |
## Uses
### Direct Use for Inference
First install the SetFit library:
```bash
pip install setfit
```
Then you can load this model and run inference.
```python
from setfit import SetFitModel
# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot-instructions_only_reasoning_1726750400.408156")
# Run inference
preds = model("Evaluation:
The answer provided is concise and directly addresses the question of whom to contact for travel reimbursement questions. It correctly refers to the email address provided in the document. The answer is well-supported by the document and does not deviate into unrelated topics.
The final evaluation: Good")
```
<!--
### Downstream Use
*List how someone could finetune this model on their own dataset.*
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Set Metrics
| Training set | Min | Median | Max |
|:-------------|:----|:---------|:----|
| Word count | 30 | 106.3538 | 221 |
| Label | Training Sample Count |
|:------|:----------------------|
| 0 | 32 |
| 1 | 33 |
### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (5, 5)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False
### Training Results
| Epoch | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0061 | 1 | 0.2332 | - |
| 0.3067 | 50 | 0.2674 | - |
| 0.6135 | 100 | 0.2116 | - |
| 0.9202 | 150 | 0.0354 | - |
| 1.2270 | 200 | 0.0036 | - |
| 1.5337 | 250 | 0.0022 | - |
| 1.8405 | 300 | 0.0017 | - |
| 2.1472 | 350 | 0.0016 | - |
| 2.4540 | 400 | 0.0015 | - |
| 2.7607 | 450 | 0.0013 | - |
| 3.0675 | 500 | 0.0013 | - |
| 3.3742 | 550 | 0.0012 | - |
| 3.6810 | 600 | 0.0012 | - |
| 3.9877 | 650 | 0.0011 | - |
| 4.2945 | 700 | 0.0012 | - |
| 4.6012 | 750 | 0.0011 | - |
| 4.9080 | 800 | 0.0011 | - |
### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.0
- Transformers: 4.44.0
- PyTorch: 2.4.1+cu121
- Datasets: 2.19.2
- Tokenizers: 0.19.1
## Citation
### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
doi = {10.48550/ARXIV.2209.11055},
url = {https://arxiv.org/abs/2209.11055},
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Efficient Few-Shot Learning Without Prompts},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--> |