max context lenght
#3
by
exorcizamus
- opened
hello. im wonderink about using this model with 65k context is it trained to handle it or its on 35k?
hello. im wonderink about using this model with 65k context is it trained to handle it or its on 35k?
It's 4k, as you can confirm here: https://huggingface.co/Undi95/Xwin-MLewd-13B-V0.2/blob/main/config.json
"max_position_embeddings": 4096,
im already using 12k and it works fine so far. so i guess i can push it till 32k since the models vocab is about that lenght