README.md

metadata

license: llama3

The official instruct model weights for "Efficient Continual Pre-training by Mitigating the Stability Gap".

Introduction

This repo contains Llama-3-Physician-8B-Instruct, a medical language model with 8 billion parameters. This model builds upon the foundation of Llama 3 and has been firstly continual pretrained on high-quality medical sub-corpus from the RefinedWeb dataset and then tuned with diverse medical and general instructions. We also use the three strategies in the paper to mitigate the stability gap during continual pretraining and instruction tuning, which boosts the model's medical task performance and reduces the computation consumption.

💻 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
model_name = "YiDuo1999/Llama-3-Physician-8B-Instruct"
device_map = 'auto'
model = AutoModelForCausalLM.from_pretrained( model_name, trust_remote_code=True,use_cache=False,device_map=device_map)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
def askme(question):
    sys_message = ''' 
    You are an AI Medical Assistant trained on a vast dataset of health information. Please be thorough and
    provide an informative answer. If you don't know the answer to a specific medical inquiry, advise seeking professional help.
    '''   
    # Create messages structured for the chat template
    messages = [{"role": "system", "content": sys_message}, {"role": "user", "content": question}]
    
    # Applying chat template
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=100, use_cache=True)
    
    # Extract and return the generated text, removing the prompt
    response_text = tokenizer.batch_decode(outputs)[0].strip()
    answer = response_text.split('<|im_start|>assistant')[-1].strip()
    return answer
# Example usage
# - Context: First describe your problem.
# - Question: Then make the question.
question = '''What is HIV?'''
print(askme(question))

the type of answer is :

HIV, or Human Immunodeficiency Virus, is a retrovirus that primarily infects cells of the human immune system, particularly CD4+ T cells, which are crucial to the body's ability to fight off infection. HIV infection can lead to AIDS, or Acquired Immune Deficiency Syndrome, a condition that causes severe damage to the immune system and makes individuals more susceptible to life-threatening infections. HIV
is transmitted through sexual contact, sharing needles, or through mother-to-child transmission during pregnancy.

🏆 Evaluation

For question-answering tasks, we have

Model	MMLU-Medical	PubMedQA	MedMCQA	MedQA-4-Option	Avg
Mistral-7B-instruct	55.8	17.8	40.2	41.1	37.5
Zephyr-7B-instruct-β	63.3	46.0	43.0	48.5	48.7
PMC-Llama-7B	59.7	59.2	57.6	49.2	53.6
Medalpaca-13B	55.2	50.4	21.2	20.2	36.7
AlpaCare-13B	60.2	53.8	38.5	30.4	45.7
BioMedGPT-LM 7B	52.0	58.6	34.9	39.3	46.2
Me-Llama-13B	-	70.0	44.9	42.7	-
Llama-3-8B instruct	82.0	74.6	57.1	60.3	68.5
JSL-Med-Sft-Llama-3-8B	83.0	75.4	57.5	74.8	72.7
GPT-3.5-turbo-1106	74.0	72.6	34.9	39.3	60.6
GPT-4	85.5	69.2	69.5	83.9	77.0
Llama-3-physician-8B instruct (ours)	80.0	76.0	80.2	60.3	74.1

For Medical claasification, relation extraction, natural language inference, summarization tasks, we have

Task type	Classification	Relation extraction	Natural Language Inference	Summarization
Datasets	HOC	DDI-2013	BioNLI	MIMIC-CXR
Mistral-7B-instruct	35.8	14.1	16.7	12.5
Zephyr-7B-instruct-β	26.1	19.4	19.9	10.5
PMC-Llama-7B	18.4	14.7	15.9	13.9
Medalpaca-13B	24.6	5.8	16.4	1.0
AlpaCare-13B	26.7	11.0	17.0	13.4
BioMedGPT-LM 7B	23.4	15.5	17.9	6.2
Me-Llama-13B	33.5	21.4	19.5	40.0
JSL-Med-Sft-Llama-3-8B	25.6	19.7	16.6	13.8
Llama-3-8B instruct	31.0	15.1	18.8	10.3
GPT-3.5-turbo-1106	54.5	21.6	31.7	13.5
GPT-4	60.2	29.2	57.8	15.2
Llama-3-physician-8B instruct (ours)	78.9	33.6	76.2	37.7

Citation

@inproceedings{Guo2024EfficientCP,
  title={Efficient Continual Pre-training by Mitigating the Stability Gap},
  author={Yiduo Guo and Jie Fu and Huishuai Zhang and Dongyan Zhao and Yikang Shen},
  year={2024},
  url={https://api.semanticscholar.org/CorpusID:270688100}
}