Missing Tokenizer.json
#5
by
tjake
- opened
I manually used the one from non-chat model but should be here too
Hi π@tjake, all the files for our base model are here: https://huggingface.co/01-ai/Yi-Coder-1.5B/tree/main
Yes that's what I used but my project depends on the tokenizer.json being checked in with each model.
Is there a reason you left it out? Would you mind adding it to the chat models?
Hi π Hello, the base model includes tokenizer.json for debugging convenience or special adjustments, while the chat model may only need tokenizer.model to improve loading and running efficiency. However, all 1.5B tokenizer.json files are the same. If your project needs it, you can download the 1.5B base tokenizer.json and add it to your project.
tjake
changed discussion status to
closed