Qwen/Qwen-VL · Cannot convert to gguf

Jan 15

I wanted to convert this model into gguf format but this did not work:
python llama.cpp/convert.py dummies/python_postprocessing/Qwen-VL --outfile Qwen-VL.gguf --outtype q8_0

Gives me this error:

/home/avi/llama.cpp/gguf-py
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00001-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00001-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00002-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00003-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00004-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00005-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00006-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00007-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00008-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00009-of-00010.bin
Loading model file dummies/python_postprocessing/Qwen-VL/pytorch_model-00010-of-00010.bin
Traceback (most recent call last):
  File "/home/avi/llama.cpp/convert.py", line 1658, in <module>
    main(sys.argv[1:])  # Exclude the first element (script name) from sys.argv
  File "/home/avi/llama.cpp/convert.py", line 1577, in main
    model_plus = load_some_model(args.model)
  File "/home/avi/llama.cpp/convert.py", line 1354, in load_some_model
    model_plus = merge_multifile_models(models_plus)
  File "/home/avi/llama.cpp/convert.py", line 782, in merge_multifile_models
    model = merge_sharded([mp.model for mp in models_plus])
  File "/home/avi/llama.cpp/convert.py", line 761, in merge_sharded
    return {name: convert(name) for name in names}
  File "/home/avi/llama.cpp/convert.py", line 761, in <dictcomp>
    return {name: convert(name) for name in names}
  File "/home/avi/llama.cpp/convert.py", line 736, in convert
    lazy_tensors: list[LazyTensor] = [model[name] for model in models]
  File "/home/avi/llama.cpp/convert.py", line 736, in <listcomp>
    lazy_tensors: list[LazyTensor] = [model[name] for model in models]
KeyError: 'transformer.wte.weight'

iHaag

Feb 24

Any luck?

alfie-ns

Jul 7

Nope gave up

perpendicularai

Jul 7

•

edited Jul 7

Try:
python llama.cpp\convert-to-hf-update.py "huggingface_token"
python llama.cpp\convert-to-hf .py "Path to Huggingface model"
python 'Path to "llama-cpp-python/examples/low_level_api/quantize.py"' "Path to ggml_fp16.gguf" "Quant option(Popular sizes: 2=q4_0, 3=q4_1)"
catch:
exception as (e):
print(e)

sanjeev-bhandari01

Jul 12

•

edited Jul 12

hi @alfie-ns ,
llama.cpp does not support the Qwen architecture LLM conversion. See issue on GitHub.