Maximum new tokens size
What is the output tokens limit for generating coherent text? At 2000 the bot writes something like random comments from the Internet
The max context size is 2048. If you generate past that it will start losing the initial context.
The max context size is 2048. If you generate past that it will start losing the initial context.
I mean "max new tokens". 200 is normal and sometimes is good (like ChatGPT answer) but 2000 is worse and incoherently at all
Then don’t use 2000 tokens. The dataset doesn’t include such long answers as far as I know. When does chatgpt give you that long of a response anyway?
Then don’t use 2000 tokens. The dataset doesn’t include such long answers as far as I know. When does chatgpt give you that long of a response anyway?
"Write a fanfic about..." XD (but this is just one example that I use most often, there are topics that require quite a detailed answer)
That’s not going to work. You’re better off using it as an ai assisted writing tool in notebook mode rather than instructing such long responses.