Appears broken, all responses about movie titles or the Turkish language

#1
by complexinteractive - opened

Regardless of sampler settings, prompt format, etc, the IQ2_S quant will only produce garbled responses about movie titles, English to Turkish translation tasks, or both. I've tried about two dozen prompts and they've all done it. The responses are clearly influenced by what it was asked (eg, "What is the capital of France" returns broken text creating fictional movie titles about Paris and then translates all of that into Turkish) so it's not straight up broken, but something is obviously wrong. I'd be very curious to hear if anyone else is having this issue.

Random prompt formats are unlikely to work, you need the correct prompt format for the model, i.e. the llama-3 instruct one. Other than that, I can't vouch for the turkish, but these are just quants of the original model, so the quality is largely that of the original model (modulo bit reduction). Try a higher quality quant if the results are not as good as expected . IQ4_XS or Q4_K_S should yield acceptable quality. Also make sure your engine actually supports llama 3 (llama.cpp was updated today only).

mradermacher changed discussion status to closed

Sign up or log in to comment