Error loading model
There is a problem loading model with llamafile version : v0.8.9
llama_model_loader: - type f32: 81 tensors
llama_model_loader: - type q5_K: 241 tensors
llama_model_loader: - type q6_K: 41 tensors
llama_model_load: error loading model: error loading model hyperparameters: invalid n_rot: 128, expected 160
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/home/andy/Projects/AI/Models/Mistral-Nemo-12B-Instruct-2407-Q5_K_M.gguf'
{"function":"load_model","level":"ERR","line":450,"model":"/home/andy/Projects/AI/Models/Mistral-Nemo-12B-Instruct-2407-Q5_K_M.gguf","msg":"unable to load model","tid":"10733792","timestamp":1721593781}
Any idea what is wrong?
Currently Mistral-Nemo isn't supported by llama.cpp as it hasn't been merged yet, see pr Mistral Nemo inference support, it's likely it'll be merged soon. If you don't want to wait you can build llama.cpp from iamlemec's mistral-nemo branch at GitHub