Update README.md

README.md

@@ -9,54 +9,59 @@ metrics:
 - bertscore
 pipeline_tag: text-classification
 ---
-# QA-Evaluation-Metrics
+# QA-Evaluation-Metrics 📊
 
 [![PyPI version qa-metrics](https://img.shields.io/pypi/v/qa-metrics.svg)](https://pypi.org/project/qa-metrics/) 
-[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/17b7vrZqH0Yun2AJaOXydYZxr3cw20Ga6?usp=sharing)
+[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Ke23KIeHFdPWad0BModmcWKZ6jSbF5nI?usp=sharing)
 
-QA-Evaluation-Metrics is a fast and lightweight Python package for evaluating question-answering models and prompting of black-box and open-source large language models. It provides various basic and efficient metrics to assess the performance of QA models. 
+> Check out the main [Repo](https://github.com/zli12321/qa_metrics)
 
-### Updates
-- Uopdated to version 0.2.17
-  - Supports prompting OPENAI GPT-series models and Claude Series models now. (Assuimg OPENAI version > 1.0)
-  - Supports prompting various open source models such as LLaMA-2-70B-chat, LLaVA-1.5 etc by calling API from [deepinfra](https://deepinfra.com/models).
-  - Added trained tiny-bert for QA evaluation. Model size is 18 MB.
-  - Pass huggingface repository name to download model directly for TransformerMatcher
+> A fast and lightweight Python package for evaluating question-answering models and prompting of black-box and open-source large language models.
 
+## 🎉 Latest Updates
 
-## Installation
-* Python version >= 3.6
-* openai version >= 1.0
+- **Version 0.2.19 Released!**
+  - Paper accepted to EMNLP 2024 Findings! 🎓
+  - Enhanced PEDANTS with multi-pipeline support and improved edge case handling
+  - Added support for OpenAI GPT-series and Claude Series models (OpenAI version > 1.0)
+  - Integrated support for open-source models (LLaMA-2-70B-chat, LLaVA-1.5, etc.) via [deepinfra](https://deepinfra.com/models)
+  - Introduced trained tiny-bert for QA evaluation (18MB model size)
+  - Added direct Huggingface model download support for TransformerMatcher
 
+## 🚀 Quick Start
 
-To install the package, run the following command:
+### Prerequisites
+- Python >= 3.6
+- openai >= 1.0
 
+### Installation
 ```bash
 pip install qa-metrics
 ```
 
-## Usage/Logistics
+## 💡 Features
 
-The python package currently provides six QA evaluation methods. 
-- Given a set of gold answers, a candidate answer to be evaluated, and a question (if applicable), the evaluation returns True if the candidate answer matches any one of the gold answer, False otherwise.
-- Different evaluation methods have distinct strictness of evaluating the correctness of a candidate answer. Some have higher correlation with human judgments than others.
-- Normalized Exact Match and Question/Answer type Evaluation are the most efficient method. They are suitable for short-form QA datasets such as NQ-OPEN, Hotpot QA, TriviaQA, SQuAD, etc.
-- Question/Answer Type Evaluation and Transformer Neural evaluations are cost free and suitable for short-form and longer-form QA datasets. They have higher correlation with human judgments than exact match and F1 score when the length of the gold and candidate answers become long.
-- Black-box LLM evaluations are closest to human evaluations, and they are not cost-free.
+Our package offers six QA evaluation methods with varying strengths:
 
-## Normalized Exact Match
-#### `em_match`
+| Method | Best For | Cost | Correlation with Human Judgment |
+|--------|----------|------|--------------------------------|
+| Normalized Exact Match | Short-form QA (NQ-OPEN, HotpotQA, etc.) | Free | Good |
+| PEDANTS | Both short & medium-form QA | Free | Very High |
+| [Neural Evaluation](https://huggingface.co/zli12321/answer_equivalence_tiny_bert) | Both short & long-form QA | Free | High |
+| [Open Source LLM Evaluation](https://huggingface.co/zli12321/prometheus2-2B) | All QA types | Free | High |
+| Black-box LLM Evaluation | All QA types | Paid | Highest |
 
-Returns a boolean indicating whether there are any exact normalized matches between gold and candidate answers.
+## 📖 Documentation
 
-**Parameters**
+### 1. Normalized Exact Match
 
-- `reference_answer` (list of str): A list of gold (correct) answers to the question.
-- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated.
+#### Method: `em_match`
+**Parameters**
+- `reference_answer` (list of str): A list of gold (correct) answers to the question
+- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated
 
 **Returns**
-
-- `boolean`: A boolean True/False signifying matches between reference or candidate answers.
+- `boolean`: True if there are any exact normalized matches between gold and candidate answers
 
 ```python
 from qa_metrics.em import em_match
@@ -64,202 +69,223 @@ from qa_metrics.em import em_match
 reference_answer = ["The Frog Prince", "The Princess and the Frog"]
 candidate_answer = "The movie \"The Princess and the Frog\" is loosely based off the Brother Grimm's \"Iron Henry\""
 match_result = em_match(reference_answer, candidate_answer)
-print("Exact Match: ", match_result)
-'''
-Exact Match:  False
-'''
 ```
 
-## F1 Score
-#### `f1_score_with_precision_recall`
-
-Calculates F1 score, precision, and recall between a reference and a candidate answer.
+### 2. F1 Score
 
+#### Method: `f1_score_with_precision_recall`
 **Parameters**
-
-- `reference_answer` (str): A gold (correct) answers to the question.
-- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated.
+- `reference_answer` (str): A gold (correct) answer to the question
+- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated
 
 **Returns**
+- `dictionary`: Contains the F1 score, precision, and recall between a gold and candidate answer
 
-- `dictionary`: A dictionary containing the F1 score, precision, and recall between a gold and candidate answer.
+#### Method: `f1_match`
+**Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `threshold` (float): F1 score threshold for considering a match (default: 0.5)
+
+**Returns**
+- `boolean`: True if F1 score exceeds threshold for any gold answer
 
 ```python
-from qa_metrics.f1 import f1_match,f1_score_with_precision_recall
+from qa_metrics.f1 import f1_match, f1_score_with_precision_recall
 
 f1_stats = f1_score_with_precision_recall(reference_answer[0], candidate_answer)
-print("F1 stats: ", f1_stats)
-'''
-F1 stats:  {'f1': 0.25, 'precision': 0.6666666666666666, 'recall': 0.15384615384615385}
-'''
-
 match_result = f1_match(reference_answer, candidate_answer, threshold=0.5)
-print("F1 Match: ", match_result)
-'''
-F1 Match:  False
-'''
 ```
 
-## Efficient and Robust Question/Answer Type Evaluation
-#### 1. `get_highest_score`
-
-Returns the gold answer and candidate answer pair that has the highest matching score. This function is useful for evaluating the closest match to a given candidate response based on a list of reference answers.
+### 3. PEDANTS
 
+#### Method: `get_score`
 **Parameters**
-
-- `reference_answer` (list of str): A list of gold (correct) answers to the question.
-- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated.
-- `question` (str): The question for which the answers are being evaluated.
+- `reference_answer` (str): A Gold answer
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
 **Returns**
+- `float`: The similarity score between two strings (0 to 1)
 
-- `dictionary`: A dictionary containing the gold answer and candidate answer that have the highest matching score.
-
-#### 2. `get_scores`
+#### Method: `get_highest_score`
+**Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
-Returns all the gold answer and candidate answer pairs' matching scores.
+**Returns**
+- `dictionary`: Contains the gold answer and candidate answer pair with highest matching score
 
+#### Method: `get_scores`
 **Parameters**
-
-- `reference_answer` (list of str): A list of gold (correct) answers to the question.
-- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated.
-- `question` (str): The question for which the answers are being evaluated.
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
 **Returns**
+- `dictionary`: Contains matching scores for all gold answer and candidate answer pairs
 
-- `dictionary`: A dictionary containing gold answers and the candidate answer's matching score.
-
-#### 3. `evaluate`
+#### Method: `evaluate`
+**Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
-Returns True if the candidate answer is a match of any of the gold answers.
+**Returns**
+- `boolean`: True if candidate answer matches any gold answer
 
+#### Method: `get_question_type`
 **Parameters**
-
-- `reference_answer` (list of str): A list of gold (correct) answers to the question.
-- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated.
-- `question` (str): The question for which the answers are being evaluated.
+- `reference_answer` (list of str): List of gold answers
+- `question` (str): The question being evaluated
 
 **Returns**
+- `list`: The type of the question (what, who, when, how, why, which, where)
 
-- `boolean`: A boolean True/False signifying matches between reference or candidate answers.
+#### Method: `get_judgement_type`
+**Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
+**Returns**
+- `list`: A list revised rules applicable to judge answer correctness
 
 ```python
 from qa_metrics.pedant import PEDANT
 
-question = "Which movie is loosley based off the Brother Grimm's Iron Henry?"
 pedant = PEDANT()
 scores = pedant.get_scores(reference_answer, candidate_answer, question)
-max_pair, highest_scores = pedant.get_highest_score(reference_answer, candidate_answer, question)
 match_result = pedant.evaluate(reference_answer, candidate_answer, question)
-print("Max Pair: %s; Highest Score: %s" % (max_pair, highest_scores))
-print("Score: %s; PANDA Match: %s" % (scores, match_result))
-'''
-Max Pair: ('the princess and the frog', 'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"'); Highest Score: 0.854451712151719
-Score: {'the frog prince': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.7131625951317375}, 'the princess and the frog': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.854451712151719}}; PANDA Match: True
-'''
-```
-
-```python
-print(pedant.get_score(reference_answer[1], candidate_answer, question))
-'''
-0.7122460127464126
-'''
 ```
 
-## Transformer Neural Evaluation
-Our fine-tuned BERT model is on 🤗 [Huggingface](https://huggingface.co/Zongxia/answer_equivalence_bert?text=The+goal+of+life+is+%5BMASK%5D.). Our Package also supports downloading and matching directly. [distilroberta](https://huggingface.co/Zongxia/answer_equivalence_distilroberta), [distilbert](https://huggingface.co/Zongxia/answer_equivalence_distilbert), [roberta](https://huggingface.co/Zongxia/answer_equivalence_roberta), and [roberta-large](https://huggingface.co/Zongxia/answer_equivalence_roberta-large) are also supported now! 🔥🔥🔥
+### 4. Transformer Neural Evaluation
 
-#### `transformer_match`
+#### Method: `get_score`
+**Parameters**
+- `reference_answer` (str): A Gold answer
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
-Returns True if the candidate answer is a match of any of the gold answers.
+**Returns**
+- `float`: The similarity score between two strings (0 to 1)
 
+#### Method: `get_highest_score`
 **Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
+
+**Returns**
+- `dictionary`: Contains the gold answer and candidate answer pair with highest matching score
 
-- `reference_answer` (list of str): A list of gold (correct) answers to the question.
-- `candidate_answer` (str): The answer provided by a candidate that needs to be evaluated.
-- `question` (str): The question for which the answers are being evaluated.
+#### Method: `get_scores`
+**Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
 **Returns**
+- `dictionary`: Contains matching scores for all gold answer and candidate answer pairs
+
+#### Method: `transformer_match`
+**Parameters**
+- `reference_answer` (list of str): List of gold answers
+- `candidate_answer` (str): Candidate answer to evaluate
+- `question` (str): The question being evaluated
 
-- `boolean`: A boolean True/False signifying matches between reference or candidate answers.
+**Returns**
+- `boolean`: True if transformer model considers candidate answer equivalent to any gold answer
 
 ```python
 from qa_metrics.transformerMatcher import TransformerMatcher
 
-question = "Which movie is loosley based off the Brother Grimm's Iron Henry?"
-# Supported models: roberta-large, roberta, bert, distilbert, distilroberta
-tm = TransformerMatcher("zli12321/answer_equivalence_roberta")
-scores = tm.get_scores(reference_answer, candidate_answer, question)
+### supports `zli12321/answer_equivalence_bert`, `zli12321/answer_equivalence_distilbert`, `zli12321/answer_equivalence_roberta`, `zli12321/answer_equivalence_distilroberta`
+tm = TransformerMatcher("zli12321/answer_equivalence_tiny_bert")
 match_result = tm.transformer_match(reference_answer, candidate_answer, question)
-print("Score: %s; bert Match: %s" % (scores, match_result))
-'''
-Score: {'The Frog Prince': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.6934309}, 'The Princess and the Frog': {'The movie "The Princess and the Frog" is loosely based off the Brother Grimm\'s "Iron Henry"': 0.7400551}}; TM Match: True
-'''
 ```
 
-## Prompting LLM For Evaluation
+### 5. LLM Integration
 
-Note: The prompting function can be used for any prompting purposes.
+#### Method: `prompt_gpt`
+**Parameters**
+- `prompt` (str): The input prompt text
+- `model_engine` (str): OpenAI model to use (e.g., 'gpt-3.5-turbo')
+- `temperature` (float): Controls randomness (0-1)
+- `max_tokens` (int): Maximum tokens in response
 
-###### OpenAI
 ```python
 from qa_metrics.prompt_llm import CloseLLM
+
 model = CloseLLM()
 model.set_openai_api_key(YOUR_OPENAI_KEY)
-prompt = 'question: What is the Capital of France?\nreference: Paris\ncandidate: The capital is Paris\nIs the candidate answer correct based on the question and reference answer? Please only output correct or incorrect.'
-model.prompt_gpt(prompt=prompt, model_engine='gpt-3.5-turbo', temperature=0.1, max_tokens=10)
-
-'''
-'correct'
-'''
+result = model.prompt_gpt(prompt=prompt, model_engine='gpt-3.5-turbo')
 ```
 
-###### Anthropic
+#### Method: `prompt_claude`
+**Parameters**
+- `prompt` (str): The input prompt text
+- `model_engine` (str): Claude model to use
+- `anthropic_version` (str): API version
+- `max_tokens_to_sample` (int): Maximum tokens in response
+- `temperature` (float): Controls randomness (0-1)
+
 ```python
 model = CloseLLM()
-model.set_anthropic_api_key(YOUR_Anthropic_KEY)
-model.prompt_claude(prompt=prompt, model_engine='claude-v1', anthropic_version="2023-06-01", max_tokens_to_sample=100, temperature=0.7)
-
-'''
-'correct'
-'''
+model.set_anthropic_api_key(YOUR_ANTHROPIC_KEY)
+result = model.prompt_claude(prompt=prompt, model_engine='claude-v1')
 ```
 
-###### deepinfra (See below for descriptions of more models)
+#### Method: `prompt`
+**Parameters**
+- `message` (str): The input message text
+- `model_engine` (str): Model to use
+- `temperature` (float): Controls randomness (0-1)
+- `max_tokens` (int): Maximum tokens in response
+
 ```python
 from qa_metrics.prompt_open_llm import OpenLLM
+
 model = OpenLLM()
 model.set_deepinfra_key(YOUR_DEEPINFRA_KEY)
-model.prompt(message=prompt, model_engine='mistralai/Mixtral-8x7B-Instruct-v0.1', temperature=0.1, max_tokens=10)
-
-'''
-'correct'
-'''
+result = model.prompt(message=prompt, model_engine='mistralai/Mixtral-8x7B-Instruct-v0.1')
 ```
 
-If you find this repo avialable, please cite our paper:
+## 🤗 Model Hub
+
+Our fine-tuned models are available on Huggingface:
+- [BERT](https://huggingface.co/Zongxia/answer_equivalence_bert)
+- [DistilRoBERTa](https://huggingface.co/Zongxia/answer_equivalence_distilroberta)
+- [DistilBERT](https://huggingface.co/Zongxia/answer_equivalence_distilbert)
+- [RoBERTa](https://huggingface.co/Zongxia/answer_equivalence_roberta)
+- [Tiny-BERT](https://huggingface.co/Zongxia/answer_equivalence_tiny_bert)
+- [RoBERTa-Large](https://huggingface.co/Zongxia/answer_equivalence_roberta-large)
+
+## 📚 Resources
+
+- [Full Paper](https://arxiv.org/abs/2402.11161)
+- [Dataset Repository](https://github.com/zli12321/Answer_Equivalence_Dataset.git)
+- [Supported Models on Deepinfra](https://deepinfra.com/models)
+
+## 📄 Citation
+
 ```bibtex
-@misc{li2024panda,
-      title={PANDA (Pedantic ANswer-correctness Determination and Adjudication):Improving Automatic Evaluation for Question Answering and Text Generation}, 
+@misc{li2024pedantspreciseevaluationsdiverse,
+      title={PEDANTS: Cheap but Effective and Interpretable Answer Equivalence}, 
       author={Zongxia Li and Ishani Mondal and Yijun Liang and Huy Nghiem and Jordan Lee Boyd-Graber},
       year={2024},
       eprint={2402.11161},
       archivePrefix={arXiv},
-      primaryClass={cs.CL}
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2402.11161}, 
 }
 ```
 
+## 📝 License
 
-## Updates
-- [01/24/24] 🔥 The full paper is uploaded and can be accessed [here](https://arxiv.org/abs/2402.11161). The dataset is expanded and leaderboard is updated.
-- Our Training Dataset is adapted and augmented from [Bulian et al](https://github.com/google-research-datasets/answer-equivalence-dataset). Our [dataset repo](https://github.com/zli12321/Answer_Equivalence_Dataset.git) includes the augmented training set and QA evaluation testing sets discussed in our paper. 
-- Now our model supports [distilroberta](https://huggingface.co/Zongxia/answer_equivalence_distilroberta), [distilbert](https://huggingface.co/Zongxia/answer_equivalence_distilbert), a smaller and more robust matching model than Bert!
-
-## License
-
-This project is licensed under the [MIT License](LICENSE.md) - see the LICENSE file for details.
+This project is licensed under the [MIT License](LICENSE.md).
 
-## Contact
+## 📬 Contact
 
-For any additional questions or comments, please contact [[email protected]].
+For questions or comments, please contact: [email protected]