ValueError: Tokenizer class BitnetTokenizer does not exist or is not currently imported.

#5
by RZJournal - opened

Updated transformers library with pip install --upgrade transformers, but still receive following error:


ValueError Traceback (most recent call last)
in <cell line: 3>()
1 from transformers import AutoTokenizer, AutoModelForCausalLM
2
----> 3 tokenizer = AutoTokenizer.from_pretrained("1bitLLM/bitnet_b1_58-3B", trust_remote_code=True)
4 model = AutoModelForCausalLM.from_pretrained("1bitLLM/bitnet_b1_58-3B")

/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
839 tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
840 if tokenizer_class is None:
--> 841 raise ValueError(
842 f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
843 )

ValueError: Tokenizer class BitnetTokenizer does not exist or is not currently imported.

Couple months later and I'm still getting this as well. Unfortunate.

The same with me :(

i am also getting this

Sign up or log in to comment