Create README.md

README.md

@@ -0,0 +1,207 @@
+---
+license: cc-by-nc-4.0
+pipeline_tag: text-generation
+tags:
+- medical
+- small LM
+- instruction-tuned
+- usmle
+- synthetic data
+---
+
+
+# Meerkat-70B (Version 1.0)
+
+🚀 Meerkat-70B is a new instruction-tuned medical AI system of the Meerkat model family.
+The model was based on the Meta's Llama-3-70B-Instruct model and fine-tuned using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. 
+This equips the model with high-level medical reasoning capabilities required for solving complex medical problems.
+For further insights into our model, please refer to our paper!
+
+📄 **Paper**: [Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks](https://arxiv.org/abs/2404.00376) 
+
+
+## Quick Start
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "dmis-lab/llama-3-meerkat-70b-v1.0"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,  # You can choose to use this when there's not enough GPU memory available.
+    device_map="auto",
+)
+
+# Multi-turn dialogue example
+messages =[
+    {"role": "system", "content": "You are a helpful doctor or healthcare professional. Guide the conversation to provide useful, complete, and scientifically-grounded answers to user questions. You have the option to compose a concise, single-turn conversation if the user's input is comprehensive to provide accurate answers. However, if essential details are missing, you should engage in a multi-turn dialogue, asking follow-up questions to gather a thorough medical history and records.\n\n"},
+    {"role": "user", "content": "Hello, doctor. I'm really concerned about my 10-year-old son. We recently discovered a painless mass in his left testicle, so we brought him to the pediatrician."},
+    {"role": "assistant", "content": "I understand your concern. Let's gather some more information. Has your son experienced any other symptoms along with the mass?"},
+    {"role": "user", "content": "Other than the mass, my son hasn't shown any symptoms. He's been his usual self, playing and eating normally."}
+]
+
+input_ids = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt"
+).to(model.device)
+
+terminators = [
+    tokenizer.eos_token_id,
+    tokenizer.convert_tokens_to_ids("<|eot_id|>")
+]
+
+outputs = model.generate(
+    input_ids,
+    max_new_tokens=1000,
+    eos_token_id=terminators,
+    do_sample=True,
+    temperature=0.7,
+)
+response = outputs[0][input_ids.shape[-1]:]
+print(tokenizer.decode(response, skip_special_tokens=True))
+```
+
+## Prompt Details
+
+To reproduce the results reported in our paper, it is advisable to utilize the identical system messages used during model training. Please refer to the guidelines detailed below.
+
+### USMLE
+
+When solving USMLE-style questions such as [MedQA](https://arxiv.org/abs/2009.13081) and [MedBullets](https://arxiv.org/abs/2402.18060), use the following system message:
+```
+messages = [
+    {"role": "system", "content": "The following is a multiple-choice question about medical knowledge. Solve this in a step-by-step fashion, starting by summarizing the available information. Output a single option from the given options as the final answer. You are strongly required to follow the specified output format; conclude your response with the phrase \"the answer is ([option_id]) [answer_string]\".\n\n"},
+    {"role": "user", "content": "Two weeks after undergoing an emergency cardiac catherization with stenting for unstable angina pectoris, a 61-year-old man has decreased urinary output and malaise. He has type 2 diabetes mellitus and osteoarthritis of the hips. Prior to admission, his medications were insulin and naproxen. He was also started on aspirin, clopidogrel, and metoprolol after the coronary intervention. His temperature is 38\u00b0C (100.4\u00b0F), pulse is 93/min, and blood pressure is 125/85 mm Hg. Examination shows mottled, reticulated purplish discoloration of the feet. Laboratory studies show:\nHemoglobin count 14 g/dL\nLeukocyte count 16,400/mm3\nSegmented neutrophils 56%\nEosinophils 11%\nLymphocytes 31%\nMonocytes 2%\nPlatelet count 260,000/mm3\nErythrocyte sedimentation rate 68 mm/h\nSerum\nUrea nitrogen 25 mg/dL\nCreatinine 4.2 mg/dL\nRenal biopsy shows intravascular spindle-shaped vacuoles. Which of the following is the most likely cause of this patient's symptoms?\" (A) Renal papillary necrosis (B) Cholesterol embolization (C) Eosinophilic granulomatosis with polyangiitis (D) Polyarteritis nodosa"},
+]
+```
+The model generates reasoning paths to solve the problem and then sequentially provides the predicted answers. 
+Since the model ends its response with "the answer is,"  it is straightforward to extract the predicted answer for comparison with the actual answer.
+
+### Multiple-choice Exams
+
+For other types of multiple-choice exams such as [MedMCQA](https://arxiv.org/abs/2203.14371) or [MMLU](https://arxiv.org/abs/2009.03300), use the following simple system message:
+```
+messages = [
+    {"role": "system", "content": "Answer the multiple-choice question about medical knowledge.\n\n"},
+    {"role": "user", "content": "In a Robertsonian translocation fusion occurs at the: (A) telomeres. (B) centromeres. (C) histones. (D) ends of the long arms."},
+]
+```
+
+### Other Use Cases
+Our model was trained using the [AlpaCare](https://github.com/xzhang97666/alpacare) instruction dataset comprising 52K examples, to enhance its generalization capabilities across diverse user prompts. 
+Feel free to design and test your prompts and to share your thoughts with us, whether the model exceeds expectations or falls short!
+
+
+## Reproducing MedQA Performance with vLLM
+
+Here is an example code for fast model evaluation in MedQA using vLLM. To adapt this code for other datasets like MedMCQA or MMLU, simply modify the instructions and update the dataset paths as needed.
+```python
+# export CUDA_VISIBLE_DEVICES=0,1
+
+import re
+from datasets import load_dataset
+from vllm import LLM, SamplingParams
+USMLE_INSTRUCTION = (
+    "The following is a multiple-choice question about medical knowledge. Solve this in"
+    " a step-by-step fashion, starting by summarizing the available information. Output"
+    " a single option from the given options as the final answer. You are strongly"
+    " required to follow the specified output format; conclude your response with the"
+    ' phrase "the answer is ([option_id]) [answer_string]".\n\n'
+)
+llm = LLM(
+    model="dmis-lab/llama-3-meerkat-70b-v1.0",
+    dtype="bfloat16",
+    gpu_memory_utilization=0.9,
+    max_model_len=2048,
+    trust_remote_code=True,
+    tensor_parallel_size=2
+)
+
+tokenizer = llm.get_tokenizer()
+
+inputs, labels = [], []
+for sample in load_dataset(
+    "GBaker/MedQA-USMLE-4-options", split="test", trust_remote_code=True
+):
+    options = sorted(sample["options"].items())
+    options = " ".join(map(lambda x: f"({x[0]}) {x[1]}", options))
+    content = tokenizer.apply_chat_template(
+        [{"role": "system", "content": USMLE_INSTRUCTION}, {"role": "user", "content": sample["question"] + " " + options}],
+        add_generation_prompt=True,
+        tokenize=False,
+    )
+    inputs.append(content)
+    labels.append(sample["answer_idx"])
+
+generated = llm.generate(
+    inputs,
+    SamplingParams(
+        temperature=0.0,
+        stop_token_ids=[tokenizer.vocab["<|eot_id|>"]],
+        max_tokens=1024,
+    ),
+)
+def extract_answer(text: str, options: str = "ABCD") -> str:
+    return (re.findall(rf"he answer is \(([{options}])\)", text) or [options[0]])[-1]
+
+correctness = []
+
+for g, l in zip(generated, labels):
+    correctness.append(extract_answer(g.outputs[0].text) == l)
+
+print(sum(correctness) / len(correctness))
+```
+
+
+## Evaluation
+
+We tested models on seven medical benchmarks: [MedQA](https://arxiv.org/abs/2009.13081), [USMLE sample test](https://www.usmle.org/prepare-your-exam), [Medbullets-4](https://arxiv.org/abs/2402.18060), [Medbullets-5](https://arxiv.org/abs/2402.18060) , [MedMCQA](https://arxiv.org/abs/2203.14371), [MMLU-Medical](https://arxiv.org/abs/2009.03300), and [JAMA Clinical Challenge](https://arxiv.org/abs/2402.18060).
+
+| **Model**                       | **Average** | **MedQA** | **USMLE** | **Medbullets-4** | **Medbullets-5** | **MedMCQA** | **MMLU-Medical** |
+|:--------------------------------|:-----------:|:---------:|:---------:|:----------------:|:----------------:|:-----------:|:----------------:|
+| GPT-4                           | 76.6        | 81.4      | 86.6      | 68.8             | 63.3             | 72.4        | **87.1**         |
+| GPT-3.5                         | 54.8        | 53.6      | 58.5      | 51.0             | 47.4             | 51.0        | 67.3             |
+| MediTron-70B (Ensemble, 5 runs) | -           | 70.2      | -         | -                | -                | 66.0        | 78.0             |
+| MediTron-7B                     | 51.0        | 50.2      | 44.6      | 51.1             | 45.5             | 57.9        | 56.7             |
+| BioMistral-7B                   | 55.4        | 54.3      | 51.4      | 52.3             | 48.7             | 61.1        | 64.6             |
+| Meerkat-7B                      | 62.6        | 70.6      | 70.3      | 58.7             | 52.9             | 60.6        | 70.5             |
+| Meerkat-8B (**New**)            | 67.3        | 74.0      | 74.2      | 62.3             | 55.5             | 62.7        | 75.2             |
+| Meerkat-70B (**New**)           | **77.9**    | **82.6**  | **87.4**  | **71.4**         | **65.3**         | **73.9**    | 86.9             |
+
+Please note that the scores in MMLU-Medical were calculated based on the average accuracies across six medical-related subjects in the original MMLU benchmark, and each result for a single subject is presented below.
+
+| **Model**                       | **Average** | **Cliniq Knowledge** | **Medical Genetics** | **Anatomy** | **Professional Medicine** | **College Biology** | **College Medicine** |
+|:--------------------------------|:-----------:|:--------------------:|:--------------------:|:-----------:|:-------------------------:|:-------------------:|:--------------------:|
+| GPT-4                           | **87.1**    | 86.4                 | **92.0**             | 80.0        | **93.8**                  | **93.8**            | 76.3                 |
+| GPT-3.5                         | 67.3        | 68.7                 | 68.0                 | 60.7        | 69.9                      | 72.9                | 63.6                 |
+| MediTron-70B (Ensemble, 5 runs) | 78.0        | 75.5                 | 85.9                 | 69.4        | 82.3                      | 86.7                | 68.0                 |
+| MediTron-7B                     | 56.7        | 57.7                 | 63.8                 | 56.9        | 56.0                      | 57.1                | 48.9                 |
+| BioMistral-7B                   | 64.6        | 59.9                 | 64.0                 | 56.5        | 60.4                      | 59.0                | 54.7                 |
+| Meerkat-7B                      | 70.5        | 71.6                 | 74.8                 | 63.2        | 77.3                      | 70.8                | 65.2                 |
+| Meerkat-8B (**New**)            | 75.2        | 74.3                 | 76.7                 | 74.8        | 75.3                      | 76.1                | 74.3                 |
+| Meerkat-70B (**New**)           | 86.9        | **87.2**             | 88.2                 | **84.4**    | 87.2                      | 87.9                | **86.6**             |
+
+
+## Reference
+
+Please see the information below to cite our paper.
+```bibtex
+@article{kim2024small,
+  title={Small language models learn enhanced reasoning skills from medical textbooks},
+  author={Kim, Hyunjae and Hwang, Hyeon and Lee, Jiwoo and Park, Sihyeon and Kim, Dain and Lee, Taewhoo and Yoon, Chanwoong and Sohn, Jiwoong and Choi, Donghee and Kang, Jaewoo},
+  journal={arXiv preprint arXiv:2404.00376},
+  year={2024}
+}
+```
+
+## Acknowledgement
+
+Research supported with Cloud TPUs from Google’s TPU Research Cloud (TRC).
+
+## Contact
+
+Feel free to email `[email protected]` if you have any questions.