asprenger

/

bloom-6b4-clp-german-instruct-lora

like 2

First, thanks a lot for this adapter, I was looking for it a while and came across this.

I was wondering if you can kindly provide (or any pointer of how to do it) code for inference? I am trying to do inference and this is my code so far

BASE_MODEL = "malteos/bloom-6b4-clp-german"

model_bloom= BloomForCausalLM.from_pretrained(
    BASE_MODEL,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)
 
tokenizer_bloom= AutoTokenizer.from_pretrained(BASE_MODEL)
model_peft = PeftModel.from_pretrained(model_bloom, "asprenger/bloom-6b4-clp-german-instruct-lora")

The last line of the code throws this error

RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
        size mismatch for base_model.model.transformer.h.0.self_attention.query_key_value.lora_A.default.weight: 
copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in current model is torch.Size([16, 
4096]).
        size mismatch for base_model.model.transformer.h.0.self_attention.query_key_value.lora_B.default.weight: 
copying a param with shape torch.Size([8192, 16, 1]) from checkpoint, the shape in current model is 
torch.Size([12288, 16]).

The error continues up to h.29

Thank you in advance :)