Tool message type raise Exception because of chat_template
I cannot add a message of type tool to give the tools call result back to the LLM since chat_template raises an Exception if type != system, user, assistant.
I had to remove the following part from the chat_template definition to make it work:
{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}\n {{- raise_exception('After the optional system message, conversation roles must alternate user/assistant/user/assistant/...') }}\n {%- endif %}
And also i had to replace this part:
{{- raise_exception('Only user and assistant roles are supported, with the exception of an initial optional system message!') }}
by:
{{- '[TOOL_RESULTS] ' + message['content'] + '[/TOOL_RESULTS]' }}
With those changes i was able to send the tools result back to the LLM.
also beams effect this output , so make sure your only using 1 beam search !
Also streaming also trips this problem ! ( something to do with sampling )
Also Use cache = true give issues !
And do not get me startted on flash attention !
For my model i had to strrip out a lot of stuff ! <<
Then get stuff from the orginal llama model and replace these in the modelling _.py file !<<
( ie they tried to implement the yarn embeddings but somehow its not correct ! )