llm-zh-tokenizer / README.md
zirui3's picture
Update README.md
fe523d6
metadata
license: cc-by-nc-4.0

Chinese tokenizer trained on Baike by using BPE algorithm