safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
I have done SFT training on the model with base model as facebook opt-350m. Later saved the trained model and trying to deploy on aws sagemaker. But I'm getting this error "safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer"
How can this error be solved in deployment
import json
from sagemaker.huggingface import HuggingFaceModel
sagemaker config
instance_type = "ml.g5.4xlarge"
number_of_gpu = 1
health_check_timeout = 300
Define Model and Endpoint configuration parameter
config = {
'HF_MODEL_ID': "/opt/ml/model", # path to where sagemaker stores the model
'SM_NUM_GPUS': json.dumps(number_of_gpu), # Number of GPU used per replica
'MAX_INPUT_LENGTH': json.dumps(1024), # Max length of input text
'MAX_TOTAL_TOKENS': json.dumps(2048), # Max length of the generation (including input text)
'HF_MODEL_QUANTIZE': "bitsandbytes",# Comment in to quantize
}
create HuggingFaceModel with the image uri
llm_model = HuggingFaceModel(
role=role,
image_uri=llm_image,
model_data=s3_model_uri,
env=config
)
Deploy model to an endpoint
https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy
llm = llm_model.deploy(
initial_instance_count=1,
instance_type=instance_type,
volume_size=400, # If using an instance with local SSD storage, volume_size must be None, e.g. p4 but not p3
container_startup_health_check_timeout=health_check_timeout, # 10 minutes to be able to load the model
)
I've tried this code for deploying. But getting the error "safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer". How can I solve it
I am not familiar with the sagemaker deployment, but google gives some information, and one is
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/10199
Thanks @ydshieh for the help.