|
--- |
|
license: mit |
|
--- |
|
## TokenMonster |
|
|
|
The documentation and code is available on Github [alasdairforsythe/tokenmonster](https://github.com/alasdairforsythe/tokenmonster). |
|
|
|
The prebuilt vocabularies are all available for download [here](https://huggingface.co/alasdairforsythe/tokenmonster/tree/main/vocabs). |
|
|
|
**July 3:** TokenMonster v1.0 has been released. The "420" prebuilt vocabularies are being released as they are completed, at a rate of around 10 per day. Let me know if there's one you want and I can prioritize it. |
|
|
|
Choose a dataset from: |
|
|
|
- `code` |
|
- `english` |
|
- `englishcode` |
|
- `fiction` |
|
|
|
Choose a vocab size from: |
|
- `1024` |
|
- `2048` |
|
- `4096` |
|
- `8000` |
|
- `16000` |
|
- `24000` |
|
- `32000` |
|
- `40000` |
|
- `50256` |
|
- `65536` |
|
- `100256` |
|
|
|
Choose an optimization mode from: |
|
- `unfiltered` |
|
- `clean` |
|
- `balanced` |
|
- `consistent` |
|
- `strict` |
|
|
|
For a capcode disabled vocabulary add: |
|
- `nocapcode` |
|
|
|
And finally add the version number: |
|
- `v1` |
|
|
|
Examples: |
|
- `fiction-24000-consistent-v1` |
|
- `code-4096-clean-nocapcode-v1` |
