Can anyone make ggml 4bit q4_0 version?
#3
by
4eJIoBek
- opened
I think it ll be usual
On the way! Got two bash windows git pushing my repos up now, might be a couple hours. I converted and quantized these with https://github.com/ggerganov/ggml/ on a MacBook pro M1 w/ 16GB RAM.
They are up! https://huggingface.co/oeathus/stablelm-7b-sft-v7-epoch-3-ggml-q4
My repo has the ggml f16 model as well as the 4 q4_X quantized versions.
Oh huge thx