Convert tiktoken to GPT2Tokenizer

#48

by abhi-db - opened Apr 10

←

Databricks org Apr 10

•

[DRAFT PR]

This PR would remove the need for tiktoken.py and convert to using GPT2Tokenizer.

abhi-db changed pull request status to open Apr 10

abhi-db changed pull request status to merged Apr 11

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment