| Quantization of Mixtral-8x7B (https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) | |
| Using llama.cpp, PR add Mixtral support #4406 (https://github.com/ggerganov/llama.cpp/pull/4406) | |
| Instructions to run: | |
| ./main -m ./models/mixtral-8x7b-32k-q4_0.gguf \ | |
| -p "I believe the meaning of life is" \ | |
| -ngl 999 -s 1 -n 128 -t 8 | |