Not following prompts

#541
by Cd957 - opened

For some reason system prompts that used to work don't anymore, the AI just doesn't follow it

Hugging Chat org

Maybe you changed models? We rotate models regularly so could be that you're now using a different one ?

@nsarrazin I can confirm this is an issue with multiple models that used to work. None of the 8 models follow the prompt anymore.

Hugging Chat org

Could you share an example, maybe an assistant that doesn't follow its prompt ?

Yup! Here's one I was using with mistralai/Mixtral-8x7B-Instruct-v0.1 :

You're an Islamic research assistant that only uses authentic Islamic texts like the Qur'an and hadith to answer questions.

It used to have an Islamic research assistant persona. For example, it would say Salam Alaykoum when I said hello.

Here's another one that used to work with mistralai/Mixtral-8x7B-Instruct-v0.1 and mistralai/Mistral-7B-Instruct-v0.3:

You're a pirate that only replies in pirate speak.

I used to type hello and it would respond with arrrgghhh...etc

Hugging Chat org

Thanks, I can reproduce the issue, investigating

Awesome, thank you!

Hugging Chat org

It seems to be a mistral issue, for example Llama 3.1 70B seems to work okay. Will check, do you know when this changed approximately ?

It's been at least 3 or 4 days since I noticed. It wasn't until I tested different prompts today and realized none of them worked.

I just tested meta-llama/Meta-Llama-3.1-70B-Instruct and it does seem to work.

Hugging Chat org

The issue seems to only be for mistralai/Mistral-7B-Instruct-v0.3 and mistralai/Mixtral-8x7B-Instruct-v0.1 which is weird since their chat template hasn't been updated for a while so I don't think anything changed...

Hugging Chat org

Foudn the issue! I'll get a fix out for mistralai/Mixtral-8x7B-Instruct-v0.1 but might take a bit longer for mistralai/Mistral-7B-Instruct-v0.3

Somewhat relevant, mistralai/Mistral-7B-Instruct-v0.3 had an update to fix the tool calling, than a follow up update to fix the chat template. Here's the commit in case haven't seen it https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/commit/b8ef225287903940a8fcc1f6e1016b29c5cc01f3

Hugging Chat org

Yup I'm asking internally about the template, that's why it might take longer 😛

No worries, I was more interested in mistralai/Mixtral-8x7B-Instruct-v0.1 :D

Thank you for looking into it!

Hugging Chat org

Should be fixed for both of them @nazimali !

nsarrazin changed discussion status to closed

Sign up or log in to comment