bad-korean-tokenizer / added_tokens.json
minseong's picture
add tokens
076a128
raw
history blame contribute delete
242 Bytes
{
"๊ฝ†๊ณ ": 50139,
"๋ฎซ์œผ๋ฉด": 50138,
"๊ฐ ํƒ†์•—์Œ": 50142,
"๋™๊ตฌ๋ ‡๊ฒŒ": 50137,
"์ด๋ผ๋Š”๋‘ฅ": 50136,
"์–ผ๋งˆ์ •๋„แ„‡แ†ž๋Š”๋ฐ": 50143,
"๊ป“๋ฑ€": 50135,
"์ฐธ์ง€์•Š๊ธฌ": 50141,
"์žฌ๋ฏบ๋‚˜์š”": 50140
}