TheBloke

/

wizard-vicuna-13B-GPTQ

like 99

Please check the README and follow the instructions for text-generation-webui. You need to set the GPTQ parameters and "save this model" and "reload this model".

Please check the README and follow the instructions for text-generation-webui. You need to set the GPTQ parameters and "save this model" and "reload this model".

This!
Or you can also load the model with these params
--wbits 4 --groupsize 128 --model_type "Llama"

Can this be used directly with AutoModelForCausalLM.from_pretrained if I dont need the text-generation-webui?
I am getting the same error.

Can this be used directly with AutoModelForCausalLM.from_pretrained if I dont need the text-generation-webui?

No. With AutoGPTQ it can be used in a manner quite similar to that. AutoGPTQ will be the future of GPTQ and supports nice easy inference from Python code. It's currently still in development and has a number of issues, so I can't blanket recommend it to you right now. But in a few days to a week I will be able to.

If you really want to do Python inference from a GPTQ model right now, check out this discussion thread https://huggingface.co/TheBloke/stable-vicuna-13B-GPTQ/discussions/1#644cc8af97a3b0904a481e3e . It has working code in it, that uses GPTQ-for-LLaMa.

I am getting the same error.

If you're still getting the error after saving the model settings, then it sounds like the model isn't installed properly. I'd suggest deleting it and following the "easy install instructions for text-generation-webui" from the beginning, and making sure you're following every step.

One irritation about oobabooga is that it makes you choose a model to load when you start it up. If the only model you have in your models directory is a model that throws an error, then you can't go in and save the parameters via the webui, because it never starts up. I think a workaround to this would be, in the models folder, create/edit the config-user.yaml file manually (which is all the UI is really doing, I believe), and add this:

wizard-vicuna-13B-GPTQ:
auto_devices: false
bf16: false
cpu: false
cpu_memory: 0
disk: false
gpu_memory_0: 0
groupsize: '128'
load_in_8bit: false
model_type: llama
pre_layer: 0
wbits: '4'

OSError: TheBloke/wizard-vicuna-13B-GPTQ does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack
error after downloading , what to do here

@a749734 Please check the README again. You need to set GPTQ parameters and then "Save settings for this model" and "reload this model"