How to use this on my local machine?
#7
by
agravans
- opened
I am new to hugging face libraries. Can someone suggest links/docs for using this model on my local or collab/kaggle notebooks since the use with transformers is not available for this model.
You can use it with llama.cpp: https://github.com/ggerganov/llama.cpp
Note, that llama.cpp is not a Python project, you need a C++ compiler and build tools (make/cmake) to build libs & examples.
https://koboldai.org/cpp is one of the easiest ways to run the ggml models, gets you a nice GUI interface to use them in.
Do note the default settings are not correct for this model, if you activate Instruct mode in the settings make sure to rename Instruction to Human and Response to Assistent.
