Tom9000/TheProfessor-155b-GUFF-Q1-v02

Update: 1.5bit quantization is now merged with the main branch of llama.cpp, so no need for dev branch to be cloned. Default dataset used for imatrix computation "wiki.train.raw" can be donwloaded from here: https://huggingface.co/datasets/ggml-org/ci

TheProfessor - 155B - 1bit GGUF

A test run of upcoming "1-bit" quantisation in llama.cpp on Eric's and AbacusAI's 155B model "TheProfessor": https://huggingface.co/abacusai/TheProfessor-155b

First run (v01) was a fail, not very coherent, so uploading only second attempt (v02). Second one seemed quite coherent, but it wasn't extensively tested.

As of of this writing, "IQ1_S" quantization is an open PR on the main branch, so to test it, dev branch would need to be cloned and compiled, instead of the main branch: https://github.com/ggerganov/llama.cpp/pull/5453

For reference, some size and perplexity comparison:

                                 Size  /  PPL  
TheProfessor-155b-Q1_S-v02.gguf   31G    9.24
TheProfessor-155b-Q4_K_M.gguf     87G    5.29

1bit quantization requires imatrix computed first. My imatrix is not really "State-Of-The-Art" by any means. There might be plenty of room for improvement, for anyone with better hardware to have a go, by trying one or all of the following:

1. Calculate imatrix from f16 or at least Q8 version of the model, instead of "Q4_K_M" (with "only" 96GB of RAM, Q4 was the largest I could load),
2. Calculate imatrix with 1000 or even more chunks instead of only 100 (100 chuncks took 8 hours on my machine),
3. Potentially use of better dataset for imatrix, instead of wikitext, might improve end results as well.

Replication steps:

1. Clone and compile "ik/iq1_s" dev branch of llama.cpp:

git clone -b ik/iq1_s https://github.com/ggerganov/llama.cpp

2. Generate imatrix file:

./imatrix -m models/TheProfessor-155b-Q4_K_M.gguf -f datasets/wiki.train.raw -o models/TheProfessor-155b-v02.imatrix --chunks 100 -b 512

3. Quantise f16 to IQ1_S:

./quantize --imatrix models/TheProfessor-155b-v02.imatrix models/TheProfessor-155b/ggml-model-f16.gguf models/TheProfessor-155b-Q1_S-v02.gguf IQ1_S