tokenizer doesn't work with the old API ?
I have used the Llama1/Llama2 inference API https://huggingface.co/docs/transformers/main/en/model_doc/llama
model = transformers.LlamaForCausalLM.from_pretrained(
"/tmp/Llama-2-7b-chat-hf",
torch_dtype=torch.bfloat16,
).cuda(
0
)
tokenizer = transformers.LlamaTokenizer.from_pretrained(
"/tmp/Llama-2-7b-chat-hf",
)
inputs = tokenizer(prompt, return_tensors="pt")
model_generation = model.generate(
inputs,
max_new_tokens=100,
do_sample=False,
return_dict_in_generate=True,
output_scores=True,
)
this code worked fine from Llama1 and Llama2. I understand that Llama3 introduced some changes on the tokenizer. when I loaded the llama3 model with the above code, at first it complains that the "tokenizer.model" is not found, I got a "tokenizer.model" from the original/ dir from the native .pth model. but then it complains that the deserialization failed, because the original/tokenizer.model file is a text file, while the llama2 tokenizer.model is a protobuf file.
so what is my best route of action? our existing code base still very much depends on the above code, it would be too complex to move to the 'pipeline' inference API now on meta3 wiki
using torch 2.0 I assume?
- => Upgrade pytorch (more)
- Or try this:
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
this_is_my_favourite_animal = "meta-llama/Meta-Llama-3-8B"
model = AutoModelForCausalLM.from_pretrained(
this_is_my_favourite_animal,
torch_dtype=torch.float16,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(this_is_my_favourite_animal)
inputs = tokenizer("in a shocking turn of events, the", return_tensors="pt").to("cuda")
model_generation = model.generate(
**inputs,
max_new_tokens=100,
do_sample=True,
return_dict_in_generate=True,
output_scores=True,
)
yes , AutoTokenizer works ! thanks