Jina AI org
edited Mar 5

Modified the tokenizer so that it adds cls tokens after every cls_token_interval number of tokens

jupyterjazz changed pull request status to open
jupyterjazz changed pull request status to closed

Sign up or log in to comment