Invalid split files?
KoboldCPP just added support for Command-R+.
At first, I tried using a joined version of IQ4xs, but that didn't boot. Then I tried using the split files as .001/002. Neither approach works in KoboldCPP v1.6.2.1. It could be Kobold, or maybe the archives are busted in some way.
Here is my error log.
Welcome to KoboldCpp - Version 1.62.1
For command line arguments, please refer to --help
***
Attempting to use CuBLAS library for faster prompt ingestion. A compatible CuBLAS will be required.
Initializing dynamic library: koboldcpp_cublas.dllNamespace(bantokens=None, benchmark=None, blasbatchsize=512, blasthreads=31, chatcompletionsadapter=None, config=None, contextsize=32768, debugmode=0, forceversion=0, foreground=False, gpulayers=31, highpriority=False, hordeconfig=None, host='', ignoremissing=False, launch=True, lora=None, mmproj=None, model=None, model_param='C:/KoboldCPP/Models/ggml-c4ai-command-r-plus-104b-iq4_xs.gguf', multiuser=1, noavx2=False, noblas=False, nocertify=False, nommap=False, noshift=False, onready='', password=None, port=5001, port_param=5001, preloadstory=None, quiet=False, remotetunnel=False, ropeconfig=[0.0, 10000.0], sdconfig=None, skiplauncher=False, smartcontext=False, ssl=None, tensor_split=None, threads=31, useclblast=None, usecublas=['normal', '0', 'mmq'], usemlock=True, usevulkan=None)
Loading model: C:\KoboldCPP\Models\ggml-c4ai-command-r-plus-104b-iq4_xs.gguf
[Threads: 31, BlasThreads: 31, SmartContext: False, ContextShift: True]The reported GGUF Arch is: command-r
Identified as GGUF model: (ver 6)
Attempting to Load...Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 |
llama_model_load: error loading model: invalid split file: C:\KoboldCPP\Models\ggml-c4ai-command-r-plus-104b-iq4_xs.ggufllama_load_model_from_file: failed to load model
Traceback (most recent call last):
File "koboldcpp.py", line 3192, in
File "koboldcpp.py", line 2942, in main
File "koboldcpp.py", line 398, in load_model
OSError: exception: access violation reading 0x0000000000000070
[24972] Failed to execute script 'koboldcpp' due to unhandled exception![process exited with code 1 (0x00000001)]
@SabinStargem
That file name doesn't look right and doesn't match what is in this repo. The splits are the *-of-* files.
I renamed them for the joining, since HJ Split doesn't like that style. It has worked with other models in the past. I will try another joining with PeaZip. (Which doesn't like -of- format either)
...But booting -of- in Kobold does work on my end. It is just that the files can't be joined, for some reason.
The AI has successfully typed out some legible text, so it looks like the implementation is an success.
You must use gguf-split to merge, please take a look at the model page I have some info about that there.