tiiuae

/

falcon-40b-instruct

like 1.17k

Follow

Technology Innovation Institute 929

Check out the special tokens near the top of the tokenizer.json file. There are special tokens for >>QUESTION<<, >>ANSWER<<, and a few other types you can play with. I've had the best results prompting it like this:

>>QUESTION<<In Python, I want to write a simple HTTP API that receives an object via POST and responds with another object. The request object contains a string prompt, float temperature, and int max_tokens. The response object contains a string response, int prompt_tokens, int completion_tokens. For now just set hard-coded values and get the response out. I'll add the logic myself afterward.
>>ANSWER<<

I haven't tried many of the other tokens out, perhaps >>INTRODUCTION<< is closest to the system prompt?

does anyone have any good prompt examples they can share or prompt templates to test this model?

according to "the bloke", the appropriate prompt to use for Falcon-instruct (which would assume how they formatted text when fine-tuning) is:
A helpful assistant who helps the user with any questions asked.
User: prompt
Assistant:

But I honestly I can't tell if that's just a best guess on his part. @TheBloke ?

I used this for reference: https://github.com/cmp-nct/ggllm.cpp/discussions/36

@cmp-nct has developed the Falcon GGML (now called GGCC) client and looked into prompt templates a lot.

Although now I look at that again, I see he doesn't show a system message, so maybe I need to check that. But User/Assistant should definitely work. He makes mention of the Falcon special tokens later in the post, which apparently do work too.

@TheBloke thanks for sharing your source!

I'm so frustrated by all of these OS models coming out with huge fan fare, but they never show sample prompts they used in fine tuning, and it's (sometimes) a nightmare trying to figure it out.

@kyleeasterly - could you please point me towards the tokenizer file? I too am confused on conventions around Falcon model prompting.

Also, what is a wizard model?

I think everyone's wrong about >>QUESTION<< and >>ANSWER<<. If you ask a >>QUESTION<<, it'll never output the >>ANSWER<< token on its own, meaning it likely wasn't trained like this.

I don't know what the right answer is, but it does seem eager to stick to a template like:

User Hi there
Mini what's up?

I've seen other users mention Mini too.