alasdairforsythe
commited on
Commit
•
492f305
1
Parent(s):
243c5b8
added english-32000, english-50256
Browse files
README.md
CHANGED
@@ -28,9 +28,9 @@ Trained models can be downloaded from here:
|
|
28 |
|-----------------|------------|--------|-------------
|
29 |
| english-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-100256.vocab)
|
30 |
| english-65536 | 65536 | UTF-8 | in-progress
|
31 |
-
| english-50256 | 50256 | UTF-8 |
|
32 |
| english-40000 | 40000 | UTF-8 | in-progress
|
33 |
-
| english-32000 | 32000 | UTF-8 |
|
34 |
| english-24000 | 24000 | UTF-8 | in-progress
|
35 |
| code-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/code-100256.vocab)
|
36 |
| code-65536 | 65536 | UTF-8 | in-progress
|
|
|
28 |
|-----------------|------------|--------|-------------
|
29 |
| english-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-100256.vocab)
|
30 |
| english-65536 | 65536 | UTF-8 | in-progress
|
31 |
+
| english-50256 | 50256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-50256.vocab)
|
32 |
| english-40000 | 40000 | UTF-8 | in-progress
|
33 |
+
| english-32000 | 32000 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-32000.vocab)
|
34 |
| english-24000 | 24000 | UTF-8 | in-progress
|
35 |
| code-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/code-100256.vocab)
|
36 |
| code-65536 | 65536 | UTF-8 | in-progress
|