teknium

/

Mistral-Trismegistus-7B

like 212

I fixed it in a different way but maybe yours is more correct

I may be wrong here, but this pr should probably be merged:

There are indeed two nested issues with added_tokens.json that prevent the model from being loaded successfully:

1. added_tokens.json is not properly json formatted. (one comma)
2. once solved there seems to be an index issue with the added tokens (the vocab size being 32000, index 1 and 2 are already taken).

When removing this file the model can be successfully loaded

I may be wrong here, but this pr should probably be merged:

There are indeed two nested issues with added_tokens.json that prevent the model from being loaded successfully:

1. added_tokens.json is not properly json formatted. (one comma)
2. once solved there seems to be an index issue with the added tokens (the vocab size being 32000, index 1 and 2 are already taken).

When removing this file the model can be successfully loaded

Deleted it