| parameters: Namespace(corpus_dir='../datasets/online_novel//data.txt', output_dir='../models/baby-chinese-llama2', model_type='bpe', max_sentence_length=4096, vocab_size=32000, max_lines=1000000, shuffle_lines=True, pad_id=3, normalization_rule_name='identity', character_coverage=0.9995, action='export') | |
| trainer_interface.cc(428) LOG(INFO) Normalizing sentences... | |
| trainer_interface.cc(537) LOG(INFO) all chars count=796343461 | |
| trainer_interface.cc(548) LOG(INFO) Done: 99.95% characters are covered. | |
| trainer_interface.cc(558) LOG(INFO) Alphabet size=5013 | |
| trainer_interface.cc(559) LOG(INFO) Final character coverage=0.9995 | |
| trainer_interface.cc(591) LOG(INFO) Done! preprocessed 800000 sentences. | |
| trainer_interface.cc(597) LOG(INFO) Tokenizing input sentences with whitespace: 800000 | |
| trainer_interface.cc(608) LOG(INFO) Done! 1021909 | |
| Raw corpus | |
| Total lines: 2995508 | |
| Total tokens: 1827.13MB | |
| Mean: 610, Median: 606.0, | |
| 5th percentile: 546.0, | |
| 25th percentile: 580.0, | |
| 75th percentile: 636.0, | |
| 95th percentile: 686.0, | |
| 99th percentile: 722.0, | |
| max: 2657 |