Tokenizer error and/or 'rope_scaling' problem
Hi,
I run into the problem below after i start the model in TGI. I use "meta-llama/Meta-Llama-3.1-8B-Instruct" as my tokenizer. Has somebody the same problem or know a solution?
The Error is:
ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}
2024-07-24T09:05:57.581856Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'PreTrainedTokenizerFast'.
The class this function is called from is 'LlamaTokenizer'.
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
what version of transformers are you using?
Be sure to update to transformers>=4.43.0
what version of transformers are you using?
Be sure to update to transformers>=4.43.0
I updated it already. After updating it i got a additional error:
if rope_scaling["type"] == "linear":
KeyError: 'type'
I also received this error even after upgrading to the most recent version of transformers...
I believe that I fixed the issue. Just FYI, this only applies if you use PIP so you might have to adapt it to your installation protocol:
If you run pip install --upgrade transformers for example, to get the latest version (currently 4.43.1) and try to run the model it'll tell you that you must install optimum.
If you try to fix this by running the typical pip install optimum, pip itself will uninstall the latest version of transformers and install transformers version 4.42.4. This is because optimum is not officially compatible with the latest version of transformers.
However, if you run pip install optimum --no-deps instead it won't uninstall the latest version of transformers.
If you run pip check it'll still show the conflict:
optimum 1.21.2 requires transformers[sentencepiece]<4.43.0,>=4.26.0, but you have transformers 4.43.1 which is incompatible.
However, my script works fine now despite this. The key is ensuring you have the latest transformers and optimum releases even though they're not officially compatible.
Keep in mind, I'm using bitsandbytes in 4-bit so if you're using other nuances this might not work.