Update byte tokenizer to be compatible with auto tokenizer and clean-up.

158e9d1

verified

pragaash commited on Feb 26