--- license: other license_name: tongyi-qianwen license_link: >- https://huggingface.co/Qwen/Qwen1.5-110B-Chat/blob/main/LICENSE model_name: Qwen1.5-110B-Chat arxiv: 2307.09288 base_model: Qwen/Qwen1.5-110B-Chat inference: false model_creator: Qwen pipeline_tag: text-generation quantized_by: Second State Inc. language: - en tags: - chat ---

# Qwen1.5-110B-Chat-GGUF ## Original Model [Qwen/Qwen1.5-110B-Chat](https://huggingface.co/Qwen/Qwen1.5-110B-Chat) ## Run with LlamaEdge - LlamaEdge version: coming soon - Prompt template - Prompt type: `chatml` - Prompt string ```text <|im_start|>system {system_message}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant ``` - Context size: `8192` - Run as LlamaEdge service ```bash wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen1.5-110B-Chat-Q2_K.gguf \ llama-api-server.wasm \ --prompt-template chatml \ --ctx-size 8192 \ --model-name qwen1.5-110b-chat ``` - Run as LlamaEdge command app ```bash wasmedge --dir .:. --nn-preload default:GGML:AUTO:Qwen1.5-110B-Chat-Q2_K.gguf \ llama-chat.wasm \ --prompt-template chatml ``` ## Quantized GGUF Models | Name | Quant method | Bits | Size | Use case | | ---- | ---- | ---- | ---- | ----- | | [Qwen1.5-110B-Chat-Q2_K.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q2_K.gguf) | Q2_K | 2 | 41.2 GB| smallest, significant quality loss - not recommended for most purposes | | [Qwen1.5-110B-Chat-Q3_K_M-00001-of-00002.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q3_K_M-00001-of-00002.gguf) | Q3_K_M | 3 | 32.2 GB| very small, high quality loss | | [Qwen1.5-110B-Chat-Q3_K_M-00002-of-00002.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q3_K_M-00002-of-00002.gguf) | Q3_K_M | 3 | 21.5 GB| very small, high quality loss | | [Qwen1.5-110B-Chat-Q4_K_M-00001-of-00003.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q4_K_M-00001-of-00003.gguf) | Q4_K_M | 4 | 32 GB| medium, balanced quality - recommended | | [Qwen1.5-110B-Chat-Q4_K_M-00002-of-00003.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q4_K_M-00002-of-00003.gguf) | Q4_K_M | 4 | 32.1 GB| medium, balanced quality - recommended | | [Qwen1.5-110B-Chat-Q4_K_M-00003-of-00003.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q4_K_M-00003-of-00003.gguf) | Q4_K_M | 4 | 3.09 GB| medium, balanced quality - recommended | | [Qwen1.5-110B-Chat-Q5_K_M-00001-of-00003.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q5_K_M-00001-of-00003.gguf) | Q5_K_M | 5 | 32.1 GB| large, very low quality loss - recommended | | [Qwen1.5-110B-Chat-Q5_K_M-00002-of-00003.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q5_K_M-00002-of-00003.gguf) | Q5_K_M | 5 | 32 GB| large, very low quality loss - recommended | | [Qwen1.5-110B-Chat-Q5_K_M-00003-of-00003.gguf](https://huggingface.co/second-state/Qwen1.5-110B-Chat-GGUF/blob/main/Qwen1.5-110B-Chat-Q5_K_M-00003-of-00003.gguf) | Q5_K_M | 5 | 14.8 GB| large, very low quality loss - recommended | *Quantized with llama.cpp b2824.*